TCP Sessions killed across VLANs

[whit3fir3]

I am running Opnsense 24.1.7 however this has been an issue for several years and I'm just now getting to the point where I'm admitting that I can't figure it out and looking for some assistance.  What is happening is a TCP connection from VLAN20 is making an SSH connection to a device in VLAN10.  The connection is established and everything works fine for about 30ish seconds and then the firewall starts blocking the connection.  This isn't unique to SSH.  I've see the same behavior with HTTP and HTTPS connections, however the issue is more easily reproducible using SSH so that is what I'll be focusing on.

Logs showing traffic being allowed both in an out on the respective VLANS and then denied after about 30ish seconds


Log Details:




Details of Block:


What I suspect is happening is happening is for some reason the firewall is VERY aggressively killing the tcp sessions.  A while back I found a setting in the Opnsense settings (don't remember what it was called) but it made the connection last longer (from 30ish seconds to around 10 minutes if I remember correctly).  Obviously is more of a band aide than a fix, so I'm wondering if anyone else has experienced issues like this or might have a clue about what's going on with my firewall?

[Saarbremer]

Hi,

to locate the problem: Does it happen to your internal VLAN connections only or are TCP connections to the outside world also affected?

Furthermore: Could you share
* your pass rules
* firewall -> settings
* the exact interface configuration of VL10 and VL20 with us? What kind of link aggregation do u use there?

[whit3fir3]

Saarbremer - Thank you for your response.  Just to clarify the issue is with traffic that is internal ONLY.  Anything routing to the internet is not impacted.  As to your other questions VL10 and VL20 are both using LACP.  I've also included a screen shot of the LACP configuration from the switch below.  Lastly, in your previous post you requested firewall settings.  Is there something specific that you'd like to see?  The only reason I ask is there are several pages of settings (General, Logging, Miscellaneous, Tuneables, Administration, etc...) and if I knew what you wanted to see I could get you exactly what you are looking for.

Pass Rule:


VL20 Configuration:


VL10 Configuration:


LAGG Configuration:


Bonding on Switch:


Thanks again for the assist and let me know if there is any other information you'd like to see

[Saarbremer]

Hi,

thanks for the info. I was interested in the firewall settings in Firewall -> Settings -> Advanced and the misc. section, esp. the firewall optimizations settings (should be normal). This tells us when states are going to be dumped after inactivity.

But then I saw your LACP interval which is... Tada... 30s

So I would suspect the LAGG setup, involved firmware, hardware or both. I am not an expert on LAGG and don't use it, so I cannot really give you more directions other than maybe https://forum.opnsense.org/index.php?topic=39630.0

You showed the live view with pass and block of the same connection. Could you provide the details to a pass AND block happening closely after? I would like to check the TCP flags.

[whit3fir3]

Saarbremer,

I really wish you had been right.  This evening I changed the LAGG from LACP to FAILOVER.  The idea being if I were to limit traffic to a single port and the problem goes away then there is 100% something going on with the LACP communication between the two device.  Sadly, the problem continues to persist even in FAILOVER mode so I don't think LACP is the issue.

After I bit more digging I did stumble across this gem - https://www.reddit.com/r/OPNsenseFirewall/comments/mcj800/tcp_connections_randomly_drop_every_30_seconds_or/.  Once I moved the pass rule to the floating rules section and set the state to sloppy I was able to keep an SSH connection active for 5 minutes.  I still need to do some more testing / investigating, cause if this has fixed it then it appears I have an asymmetric routing issue.

Thanks again for the help

[Saarbremer]

Hi,

thanks for sharing. Going with floating rules on LAGG is sth new to me. It would explain the odd behaviour of your observations, though. But again, LAGG has not been my business, yet.

Let's see what FreeBSD 14 brings to us.

[netnut]

Sadly, the problem continues to persist even in FAILOVER mode so I don't think LACP is the issue.

If it's related to your issue or not, you can still optimize your current LAGG setup a bit without any costs or drawbacks:

OPNsense LAGG Interface:

* Check the "Fast Timeout" checkbox
* From the multiselect dropdown menu "Hash Layers", check both L2 and L3

Switch Bonding Interface

* Change LACP Rate from 30s (slow) to 1s (fast).
* Transmit Hash Policy: layer2+3 (which should be the Linux default xmit_hash_policy for the bonding module in 802.3ad / LACP mode)

Those LACP settings should always match on both sides, your screenshot doesn't display any Hash Layers configured on the OPNsense LAGG interface.