OPNsense and TrueNAS Inter-VLAN Routing

[felippe]

Hello all,

In the attached file is the network diagram of my home network. Some details which may help in identifying a solution to the problem I am experiencing:

- the focus is on "opnsense" - "switch" - "truenas"
- the VLANs and interfaces of interest are:
VLAN 1 (MGMT) tagged on em0 (opnsense), port 10 (switch), lagg0 (truenas), BMC (truenas)
VLAN 2 (USERS) tagged on same ports as VLAN 1 less BMC, and untagged on port 6 (switch)
- truenas: VLAN 1 (MGMT) IP address:
192.168.1.2 tagged on BMC (IPMI interface)
192.168.1.3 tagged on lagg0 (igb0 and igb1 LACP link aggregation)
- truenas: VLAN 2 (USERS) IP address:
192.168.2.3 tagged on lagg0 (igb0 and igb1 link aggregation)
- opnsense: VLAN 1 (MGMT) IP address: 192.168.1.1 tagged on em0
- opnsense: VLAN 2 (USERS) IP address: 192.168.2.1 tagged on em0
- rpi4: IP address: 192.168.2.21
- switch: VLAN 1 (MGMT) IP address: 192.168.1.4 tagged on ports 10, Link Aggregation 1 (ports 3 and 4, LACP), and 5
- switch: VLAN 2 (USERS) tagged on ports 10 and Link Aggregation 1, and untagged on port 6

The only firewall rules configured on opnsense are:
[MGMT] Pass | Protocol IPV4 * | Source: MGMT Net | Source Port * | Destination: * | Dest. Port * | Gateway * | Description: Allow all
[USERS] Pass | Protocol IPV4 * | Source: USERS Net | Source Port * | Destination: * | Dest. Port * | Gateway * | Description: Allow all

The problem:

- access the web UI of truenas from rpi4 web browser on 192.168.1.3 and
- truenas SSH access from rpi4: $ ssh root@192.168.1.3
HTTP / HTTPS connection drops after less then a minute, then restore, drops again and so on; SSH connection drops and, obviously, doesn't restore without me entering the command again.

I do not experience this issues when using the 192.168.2.3 IP address. Even more: no lost connectivity when assigning a static IP to rpi4 in VLAN 1 [MGMT] (ex. 192.168.1.10). So everything works fine when both truenas and rpi4 are in the same network.

And, no issues when accessing the web UI for the IPMI interface on 192.168.1.2 (VLAN 1) from rpi4 with an IP address on VLAN 2 - so this time, no inter-VLAN routing issues.

Would this be an opnsense routing issue or truenas link aggregation one? The next step in troubleshooting will be to "break" the link aggregation and see if the problem persists when using a standard link, but I would like to have the community's feedback first. Just to add that everything was working fine when I had another VLAN configured on all devices (VLAN 3), but then I decided to get rid of it and simplify the design by bringing those devices in VLAN 2.

Your input will be appreciated.

[shelter_]

Hi felippe,

Did you ever find a solution to this problem?  I have a very similar issue of freezing SSH sessions and lost HTTPS connections when traversing networks, and when the destination device has multiple VLANs.  I also had LACP connections and broke them apart to individual VLANs on individual NICs, this did not solve my problem.  I've been working on trying to solve this problem for about a month now.  I don't want to hijack your post with my own details, but I am interested in if/how you solved your own problem.

[Patrick M. Hausen]

First: make sure to disable hardware offloading on all physical interfaces involved, on both your TruNAS and your OPNsense.

[Isopote42]

Hello,

I am responding to this old post for people like me who found it after a Google search. I have encountered the same error on HTTPS Web UI and SSH but have found a solution and can now share it.

I don't understand why, but stateful connections do not work properly.

Follow these instructions to work around the issue:

In "Firewall: Rules: LAN," edit your rule that allows connections between zones.
Click on the "Advanced Features" button and search for "State Type."
Set it to "none" instead of "keep state."

For more information about stateful connections, you can visit:
https://learningnetwork.cisco.com/s/question/0D53i00000Ksup8CAB/stateful-firewall-overview

Best regards,