[solved] pf ruleset causing client to get kicked off of TrueNAS web UI periodically

[OPNenthu]

Hello all, I'm wrestling with a first TrueNAS deployment and this issue is beyond my skillset.  Asking here because I think it's relevant to pf/OPNsense and maybe my particular rules causing the issue.

The NAS has two integrated NICs, an Intel I226-V (top) and an Aquantia AQC113 (bottom):

Code Select Expand

truenas_admin@truenas[~]$ ethtool -i enp6s0
driver: igc
version: 6.12.33-production+truenas
firmware-version: 2014:8877
expansion-rom-version:
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


Code Select Expand

truenas_admin@truenas[~]$ ethtool -i enp3s0
driver: atlantic
version: 6.12.33-production+truenas
firmware-version: 1.3.31
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no

At first I configured the management interface on enp6s0 with a static IP address, as per TrueNAS guidelines.  This interface has IP 192.168.1.118 on LAN.  It has the default gateway 192.168.1.1, DNS also 192.168.1.1, and the web UI is bound to this IP only.

Up to this point everything is working with a stable connection.  I'm able to access the web UI from a client on VLAN 30 ("CLEAR net") using a Floating rule with direction IN on the CLEAR interface that allows the client to "any" on LAN.

The problem comes when I configure the second interface.  I assign it a static IP 172.21.30.118 which is from the CLEAR subnet as this will be my data interface for SMB shares (the SMB service will bind to it).  I don't assign any additional gateway or DNS, and no static routes.  Just the IP on the interface.  IPv6 auto-config is still disabled in TrueNAS.

After this step, my TrueNAS web UI becomes unstable.  I'm still able to log in, but after some seconds I'm kicked off.  This process repeats itself ad-infinitum and I'm not able to stay in the UI long enough to get work done.  This dialog appears in between when I'm kicked off and the login prompt follows it.

You cannot view this attachment.

If I remove the IP on the second interface, then everything becomes stable again.

Checking the firewall logs while this happens I can see the initial request is allowed as it's matching the Floating rule, and it gets a response.  I don't know why the DNS lookups in between are logged at all as those are within the same subnet (this is a group rule).  Finally another interface group rule is matching instead which is rejecting and causing me to get kicked off.

You cannot view this attachment.

This happens in a loop as mentioned above.  I can see that the source ports changed in between the login and the rejected packets.  This also repeats and is reflected in the state table with new states being generated each cycle.

Also, there is a group 'pass' rule which is supposed to be overriding this 'reject' rule in any case, but that's not happening.

Floating:
You cannot view this attachment.

Group rules (from the CLEAR interface's view):
You cannot view this attachment.

The "PORTS_OUT_LAN" alias already has port 443 included.

Did I set up some kind of asymmetric routing condition when I configured the second NIC?  Is this a TrueNAS quirk? 

I do have Proxmox management on the same LAN network and with a separate VM bridge, but no problems at all there.  That's been stable for a while now.

TrueNAS ip/route info:

Code Select Expand

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:xx:xx:xx:41:0a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.118/24 brd 192.168.1.255 scope global enp6s0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:410a/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:xx:xx:xx:41:09 brd ff:ff:ff:ff:ff:ff
    inet 172.21.30.118/24 brd 172.21.30.255 scope global enp3s0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:4109/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever


Code Select Expand

truenas_admin@truenas[~]$ route  
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         firewall        0.0.0.0         UG    0      0        0 enp6s0
172.21.30.0     0.0.0.0         255.255.255.0   U     0      0        0 enp3s0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 enp6s0


[Patrick M. Hausen]

TrueNAS does not support policy routing so if you have network interfaces in different networks it will always answer any client from

- the directly connected one, if present
- the one with the default gateway, otherwise

Specifically there is no separation of the UI and the file sharing services. Not possible, don't try it, you will fail.

All of TrueNAS runs on a single IP stack. Only VMs can be connected to a separate network via a bridge (without an IP address on the TrueNAS host for that bridge). Also allegedly for these "experimental" LXC containers.

But, repeating myself, not for any TN services. UI, SMB, NFS ... all share the same network stack. Even "apps" do. While you can set a separate IP for apps for ingress, they will always use the main NAS IP for egress.


HTH,
Patrick

[OPNenthu]

Ayyy, this is a painful lesson.  I spent a couple days. :)

Thank you, Patrick.

[Patrick M. Hausen]

It's the standard behaviour for IP routing. Specifically in IP there is no "connection" anywhere. Routing decisions are made per packet based on destination address only.

Also the information on which interface some packet arrived is not kept, anywhere. So the reply can never be based on "but the request came in on this and that interface - the reply must go out the same one!" No. Does not exist and never did. IP is about packet switching as opposed to curcuit switching.

I have been fighting uphill battles with folks who just would not accept the world does not work how they think it does over on the TrueNAS forum.

Yes, one can implement systems that work like that. Of course. But it's not how any Unix/Linux based implementation works by default. And it would be proprietary, there's no standard mandating anything but routing based on destination address.

[OPNenthu]

It helps me to write this out and make sure I got the full picture-

I'm not seeing the issue in Proxmox NOT because it is designed differently than TrueNAS.  It's because there is no non-default return path for management traffic.  Management requests always go to the configured interface with the default IP and gateway and reply from the same, because I don't have any interface with IPs on direct connected networks.

The VM/container bridge, even though it's on a separate interface, cannot respond anyway as it's unconfigured.  Because it's VLAN-aware, VMs/LXCs that are tagged on the bridge are isolated to the respective networks.

Similarly, TrueNAS VMs/LXCs can go on a bridge with VLANs as long as the bridge, its VLAN parents, and the parent IF are all unconfigured.  It might need to be separate bridges in TN, I think, because there is no concept of a VLAN-aware bridge but I'm not certain.  (The Aquantia NIC falls down in a layered scenario in my limited testing so far; seems like immature firmware/drivers from other reports).

---

Up to this point things make sense.

Where I'm getting tripped up still is that TrueNAS gives options for binding the UI and the services separately, and there are videos showing that TrueNAS can have different networks for iSCSI (such as for a SAN for Windows hypervisors) and for shares for office clients, for example.

I can imagine that they have a NIC on each network and bind the respective service to an IP on it, so in that case the service is directly connected on the respective subnet.  There should be no routing for Hypervisor<->iSCSI or for client<->SMB.

Where does the management interface go?  Is it available to all and locked down only with strong authentication? 

Is the purpose of allowing the UI to be bound on a specific IP just so that people can't find it by typing in https://truenas.local?

---

I found one such thread on the TN forum that was funny.  After you explained the limitation, the guy was upset and came up with an elaborate scheme involving intermediate switches and routing / firewall tricks to get his isolated management.  He never updated to say if he had success, lol.

[Patrick M. Hausen]

You can bind different services to different interfaces, of course. But if your management desktop and TN share a common network and you define "management" to be a different one to be accessed through a firewall, TN will send the replies through the common network bypassing the firewall, because that's how routing works.

A separate storage network for e.g. iSCSI assumes that all clients and the TN server share that network, so no asymmetric routing occurs. That of course is perfectly reasonable. Same for e.g. NFS for VMware.

But placing mangement in a separate network does not work unless the management station is in that same network. Just like with the storage examples.

And yes, with Proxmox (or ESXi) the host usually does not have IP addresses in those bridge/vSwitch networks used for VMs or containers. And you can do the same with TN.

Only services running on the host all share a single stack. Which never happens with Proxmox because Proxmox does not offer file sharing.

[OPNenthu]

I was using the floating rule as a lazy shortcut to not have to change switch ports but now I see that it doesn't even work except in specific circumstances.

[OPNenthu]

Thanks again, Patrick.  This little bit of new understanding made all the difference and it seems I may have misjudged the AQC113 NIC earlier.  I got the layered setup working, finally.

Since they're asymmetrical I bonded them in active/backup with the 10GbE member as primary.  Then I added br1 for native access (in case I want some VMs or UniFi OS on there later) and assigned it the main IP.  Then added a few VLANs on the bond and separate bridges for each VLAN.  Finally assigned an IP to br30 and bound the web UI to this as well (at least temporarily), so now I'm able to access it from my client network for configuration changes without any hiccups.

Code Select Expand

truenas_admin@truenas[~]$ ip -brief a                   
lo               UNKNOWN        127.0.0.1/8 ::1/128
enp6s0           UP            
enp3s0           UP            
bond1            UP             fe80::<redacted>:4109/64
vlan20@bond1     UP             fe80::<redacted>:4109/64
vlan30@bond1     UP             fe80::<redacted>:4109/64
vlan60@bond1     UP             fe80::<redacted>:4109/64
br1              UP             192.168.1.118/24 fe80::<redacted>:850b/64
br20             UP             fe80::<redacted>:9746/64
br30             UP             172.21.30.118/24 fe80::<redacted>:435e/64
br60             UP             fe80::<redacted>:952c/64

If I did this correctly with the separate bridges, then there shouldn't be any RA spillover when I enable IPv6 auto-config in TN.

I don't have a 10GbE client to test with, but I'm at least able to saturate a 2.5GbE link from my client to the NAS using iperf3:

Code Select Expand

$ iperf3 -c truenas.clear.h1.internal
Connecting to host truenas.clear.h1.internal, port 5201
[  5] local 172.21.30.100 port 46912 connected to 172.21.30.118 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   283 MBytes  2.37 Gbits/sec    0    311 KBytes      
[  5]   1.00-2.00   sec   281 MBytes  2.36 Gbits/sec    0    291 KBytes      
[  5]   2.00-3.00   sec   281 MBytes  2.36 Gbits/sec    0    297 KBytes      
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec    0    303 KBytes      
[  5]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec    0    294 KBytes      
[  5]   5.00-6.00   sec   281 MBytes  2.36 Gbits/sec    0    300 KBytes      
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec    0    291 KBytes      
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec    0    291 KBytes      
[  5]   8.00-9.00   sec   281 MBytes  2.36 Gbits/sec    0    294 KBytes      
[  5]   9.00-10.00  sec   281 MBytes  2.35 Gbits/sec    0    291 KBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.74 GBytes  2.36 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec                  receiver

Look like the AQC113 is so far not falling down :)  Hope this keeps up.

[nero355]

TrueNAS does not support policy routing so if you have network interfaces in different networks it will always answer any client from

- the directly connected one, if present
- the one with the default gateway, otherwise

Specifically there is no separation of the UI and the file sharing services. Not possible, don't try it, you will fail.

What is TrueNAS based on these days ?!

First it was FreeBSD and then they switched to Linux as far as I know ?

Sounds like very bad decisions were made the last couple of years if it for example doesn't have SystemD and it's Policy Routing/Source Based Routing onboard like you would expect it to have...

You can bind different services to different interfaces, of course. But if your management desktop and TN share a common network and you define "management" to be a different one to be accessed through a firewall, TN will send the replies through the common network bypassing the firewall, because that's how routing works.

I had that issue (A-Symmetrical Routing blocked by the OPNsense Default Block Rule) after switching from my old Ubiquiti USG 3P Router to OPNsense and solved it via SystemD Network Configuration for :
eth0
eth0.10 VLAN
eth0.any other VLAN

Now my Traffic is no longer blocked when accessing this Multi-VLAN Interface device from VLAN 10 to it's Management eth0 address and my SSH connection for example doesn't time out! :)

A separate storage network for e.g. iSCSI assumes that all clients and the TN server share that network, so no asymmetric routing occurs. That of course is perfectly reasonable. Same for e.g. NFS for VMware.

As long as you keep things 1:1 connected in the same subnet there are no issues indeed :)

But placing mangement in a separate network does not work unless the management station is in that same network. Just like with the storage examples.

Unless you can manipulate the Routing Table on the device you are accessing ofcourse!

it seems I may have misjudged the AQC113 NIC earlier.

Look like the AQC113 is so far not falling down :)

Hope this keeps up.

I hope so too for you, because I have just read your other topic about the NIC and I am really disappointed that the driver support is not all that great/how it should be or could have been...

It was soo promising when it was announced many years ago : Just € 95 for a 1 port 10 Gbps NIC with RJ45 was WOW!!! at the time!

[OPNenthu]

I did also drop a note in the Hardware section topic to correct the record :)