My pf ruleset causing client to get kicked off of TrueNAS web UI periodically

[OPNenthu]

Hello all, I'm wrestling with a first TrueNAS deployment and this issue is beyond my skillset.  Asking here because I think it's relevant to pf/OPNsense and maybe my particular rules causing the issue.

The NAS has two integrated NICs, an Intel I226-V (top) and an Aquantia AQC113 (bottom):

Code Select Expand

truenas_admin@truenas[~]$ ethtool -i enp6s0
driver: igc
version: 6.12.33-production+truenas
firmware-version: 2014:8877
expansion-rom-version:
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


Code Select Expand

truenas_admin@truenas[~]$ ethtool -i enp3s0
driver: atlantic
version: 6.12.33-production+truenas
firmware-version: 1.3.31
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no

At first I configured the management interface on enp6s0 with a static IP address, as per TrueNAS guidelines.  This interface has IP 192.168.1.118 on LAN.  It has the default gateway 192.168.1.1, DNS also 192.168.1.1, and the web UI is bound to this IP only.

Up to this point everything is working with a stable connection.  I'm able to access the web UI from a client on VLAN 30 ("CLEAR net") using a Floating rule with direction IN on the CLEAR interface that allows the client to "any" on LAN.

The problem comes when I configure the second interface.  I assign it a static IP 172.21.30.118 which is from the CLEAR subnet as this will be my data interface for SMB shares (the SMB service will bind to it).  I don't assign any additional gateway or DNS, and no static routes.  Just the IP on the interface.  IPv6 auto-config is still disabled in TrueNAS.

After this step, my TrueNAS web UI becomes unstable.  I'm still able to log in, but after some seconds I'm kicked off.  This process repeats itself ad-infinitum and I'm not able to stay in the UI long enough to get work done.  This dialog appears in between when I'm kicked off and the login prompt follows it.

You cannot view this attachment.

If I remove the IP on the second interface, then everything becomes stable again.

Checking the firewall logs while this happens I can see the initial request is allowed as it's matching the Floating rule, and it gets a response.  I don't know why the DNS lookups in between are logged at all as those are within the same subnet (this is a group rule).  Finally another interface group rule is matching instead which is rejecting and causing me to get kicked off.

You cannot view this attachment.

This happens in a loop as mentioned above.  I can see that the source ports changed in between the login and the rejected packets.  This also repeats and is reflected in the state table with new states being generated each cycle.

Also, there is a group 'pass' rule which is supposed to be overriding this 'reject' rule in any case, but that's not happening.

Floating:
You cannot view this attachment.

Group rules (from the CLEAR interface's view):
You cannot view this attachment.

The "PORTS_OUT_LAN" alias already has port 443 included.

Did I set up some kind of asymmetric routing condition when I configured the second NIC?  Is this a TrueNAS quirk? 

I do have Proxmox management on the same LAN network and with a separate VM bridge, but no problems at all there.  That's been stable for a while now.

TrueNAS ip/route info:

Code Select Expand

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:xx:xx:xx:41:0a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.118/24 brd 192.168.1.255 scope global enp6s0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:410a/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 6c:xx:xx:xx:41:09 brd ff:ff:ff:ff:ff:ff
    inet 172.21.30.118/24 brd 172.21.30.255 scope global enp3s0
       valid_lft forever preferred_lft forever
    inet6 fe80::xxxx:xxxx:xxxx:4109/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever


Code Select Expand

truenas_admin@truenas[~]$ route   
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         firewall        0.0.0.0         UG    0      0        0 enp6s0
172.21.30.0     0.0.0.0         255.255.255.0   U     0      0        0 enp3s0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 enp6s0


[Patrick M. Hausen]

TrueNAS does not support policy routing so if you have network interfaces in different networks it will always answer any client from

- the directly connected one, if present
- the one with the default gateway, otherwise

Specifically there is no separation of the UI and the file sharing services. Not possible, don't try it, you will fail.

All of TrueNAS runs on a single IP stack. Only VMs can be connected to a separate network via a bridge (without an IP address on the TrueNAS host for that bridge). Also allegedly for these "experimental" LXC containers.

But, repeating myself, not for any TN services. UI, SMB, NFS ... all share the same network stack. Even "apps" do. While you can set a separate IP for apps for ingress, they will always use the main NAS IP for egress.


HTH,
Patrick

[OPNenthu]

Ayyy, this is a painful lesson.  I spent a couple days. :)

Thank you, Patrick.

[Patrick M. Hausen]

It's the standard behaviour for IP routing. Specifically in IP there is no "connection" anywhere. Routing decisions are made per packet based on destination address only.

Also the information on which interface some packet arrived is not kept, anywhere. So the reply can never be based on "but the request came in on this and that interface - the reply must go out the same one!" No. Does not exist and never did. IP is about packet switching as opposed to curcuit switching.

I have been fighting uphill battles with folks who just would not accept the world does not work how they think it does over on the TrueNAS forum.

Yes, one can implement systems that work like that. Of course. But it's not how any Unix/Linux based implementation works by default. And it would be proprietary, there's no standard mandating anything but routing based on destination address.