My pf ruleset causing client to get kicked off of TrueNAS web UI periodically

[OPNenthu]

Hello all, I'm wrestling with a first TrueNAS deployment and this issue is beyond my skillset.  Asking here because I think it's relevant to pf/OPNsense and maybe my particular rules causing the issue.

The NAS has two integrated NICs, an Intel I226-V (top) and an Aquantia AQC113 (bottom):

Code Select Expand

truenas_admin@truenas[~]$ ethtool -i enp6s0
driver: igc
version: 6.12.33-production+truenas
firmware-version: 2014:8877
expansion-rom-version:
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


Code Select Expand

truenas_admin@truenas[~]$ ethtool -i enp3s0
driver: atlantic
version: 6.12.33-production+truenas
firmware-version: 1.3.31
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no

At first I configured the management interface on enp6s0 with a static IP address, as per TrueNAS guidelines.  This interface has IP 192.168.1.118 on LAN.  It has the default gateway 192.168.1.1, DNS also 192.168.1.1, and the web UI is bound to this IP only.

Up to this point everything is working with a stable connection.  I'm able to access the web UI from a client on VLAN 30 ("CLEAR net") using a Floating rule with direction IN on the CLEAR interface that allows the client to "any" on LAN.

The problem comes when I configure the second interface.  I assign it a static IP 172.21.30.118 which is from the CLEAR subnet as this will be my data interface for SMB shares (the SMB service will bind to it).  I don't assign any additional gateway or DNS, and no static routes.  Just the IP on the interface.  IPv6 auto-config is still disabled in TrueNAS.

After this step, my TrueNAS web UI becomes unstable.  I'm still able to log in, but after some seconds I'm kicked off.  This process repeats itself ad-infinitum and I'm not able to stay in the UI long enough to get work done.  This dialog appears in between when I'm kicked off and the login prompt follows it.

You cannot view this attachment.

If I remove the IP on the second interface, then everything becomes stable again.

Checking the firewall logs while this happens I can see the initial request is allowed as it's matching the Floating rule, and it gets a response.  I don't know why the DNS lookups in between are logged at all as those are within the same subnet (this is a group rule).  Finally another interface group rule is matching instead which is rejecting and causing me to get kicked off.

You cannot view this attachment.

This happens in a loop as mentioned above.  I can see that the source ports changed in between the login and the rejected packets.  This also repeats and is reflected in the state table with new states being generated each cycle.

Also, there is a group 'pass' rule which is supposed to be overriding this 'reject' rule in any case, but that's not happening.

Floating:
You cannot view this attachment.

Group rules (from the CLEAR interface's view):
You cannot view this attachment.

The "PORTS_OUT_LAN" alias already has port 443 included.

Did I set up some kind of asymmetric routing condition when I configured the second NIC?  Is this a TrueNAS quirk? 

I do have Proxmox management on the same LAN network and with a separate VM bridge, but no problems at all there.  That's been stable for a while now.

TrueNAS ip/route info:

Code Select Expand

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:xx:xx:xx:41:0a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.118/24 brd 192.168.1.255 scope global enp6s0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:410a/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:xx:xx:xx:41:09 brd ff:ff:ff:ff:ff:ff
    inet 172.21.30.118/24 brd 172.21.30.255 scope global enp3s0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:4109/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever


Code Select Expand

truenas_admin@truenas[~]$ route   
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         firewall        0.0.0.0         UG    0      0        0 enp6s0
172.21.30.0     0.0.0.0         255.255.255.0   U     0      0        0 enp3s0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 enp6s0


[Patrick M. Hausen]

TrueNAS does not support policy routing so if you have network interfaces in different networks it will always answer any client from

- the directly connected one, if present
- the one with the default gateway, otherwise

Specifically there is no separation of the UI and the file sharing services. Not possible, don't try it, you will fail.

All of TrueNAS runs on a single IP stack. Only VMs can be connected to a separate network via a bridge (without an IP address on the TrueNAS host for that bridge). Also allegedly for these "experimental" LXC containers.

But, repeating myself, not for any TN services. UI, SMB, NFS ... all share the same network stack. Even "apps" do. While you can set a separate IP for apps for ingress, they will always use the main NAS IP for egress.


HTH,
Patrick

[OPNenthu]

Ayyy, this is a painful lesson.  I spent a couple days. :)

Thank you, Patrick.

[Patrick M. Hausen]

It's the standard behaviour for IP routing. Specifically in IP there is no "connection" anywhere. Routing decisions are made per packet based on destination address only.

Also the information on which interface some packet arrived is not kept, anywhere. So the reply can never be based on "but the request came in on this and that interface - the reply must go out the same one!" No. Does not exist and never did. IP is about packet switching as opposed to curcuit switching.

I have been fighting uphill battles with folks who just would not accept the world does not work how they think it does over on the TrueNAS forum.

Yes, one can implement systems that work like that. Of course. But it's not how any Unix/Linux based implementation works by default. And it would be proprietary, there's no standard mandating anything but routing based on destination address.

[OPNenthu]

It helps me to write this out and make sure I got the full picture-

I'm not seeing the issue in Proxmox NOT because it is designed differently than TrueNAS.  It's because there is no non-default return path for management traffic.  Management requests always go to the configured interface with the default IP and gateway and reply from the same, because I don't have any interface with IPs on direct connected networks.

The VM/container bridge, even though it's on a separate interface, cannot respond anyway as it's unconfigured.  Because it's VLAN-aware, VMs/LXCs that are tagged on the bridge are isolated to the respective networks.

Similarly, TrueNAS VMs/LXCs can go on a bridge with VLANs as long as the bridge, its VLAN parents, and the parent IF are all unconfigured.  It might need to be separate bridges in TN, I think, because there is no concept of a VLAN-aware bridge but I'm not certain.  (The Aquantia NIC falls down in a layered scenario in my limited testing so far; seems like immature firmware/drivers from other reports).

---

Up to this point things make sense.

Where I'm getting tripped up still is that TrueNAS gives options for binding the UI and the services separately, and there are videos showing that TrueNAS can have different networks for iSCSI (such as for a SAN for Windows hypervisors) and for shares for office clients, for example.

I can imagine that they have a NIC on each network and bind the respective service to an IP on it, so in that case the service is directly connected on the respective subnet.  There should be no routing for Hypervisor<->iSCSI or for client<->SMB.

Where does the management interface go?  Is it available to all and locked down only with strong authentication? 

Is the purpose of allowing the UI to be bound on a specific IP just so that people can't find it by typing in https://truenas.local?

---

I found one such thread on the TN forum that was funny.  After you explained the limitation, the guy was upset and came up with an elaborate scheme involving intermediate switches and routing / firewall tricks to get his isolated management.  He never updated to say if he had success, lol.

[Patrick M. Hausen]

You can bind different services to different interfaces, of course. But if your management desktop and TN share a common network and you define "management" to be a different one to be accessed through a firewall, TN will send the replies through the common network bypassing the firewall, because that's how routing works.

A separate storage network for e.g. iSCSI assumes that all clients and the TN server share that network, so no asymmetric routing occurs. That of course is perfectly reasonable. Same for e.g. NFS for VMware.

But placing mangement in a separate network does not work unless the management station is in that same network. Just like with the storage examples.

And yes, with Proxmox (or ESXi) the host usually does not have IP addresses in those bridge/vSwitch networks used for VMs or containers. And you can do the same with TN.

Only services running on the host all share a single stack. Which never happens with Proxmox because Proxmox does not offer file sharing.

[OPNenthu]

I was using the floating rule as a lazy shortcut to not have to change switch ports but now I see that it doesn't even work except in specific circumstances.