Is sloppy state tracking a bad thing?

[FlyveHest]

Hi all

I am in the process of migrating from an old Edgerouter 4 to OPNSense, and for the most part, i've gotten things working as intended, but, I currently have a single pain-point that I am not entirely sure about.

I am currently using 3 ports on my device, 0 is WAN (which I do not think is relevant for this), I have a routed /29 on a server in port 1, and then I have my LAN on port 2.

When I connect to a TCP based service on my server from my LAN (port 2 to port 1), I can connect just fine, so the firewall rule work, but after 15 or so seconds, I start seeing "Default deny / state violation rule" in the firewall log, and then the session is disconnected.

But, if I set State Type to sloppy state, then it works as intended.

The docs does not say much about sloppy state tracking, so my question is, is it a "bad" thing to enable?

Or is it just something that is required, maybe due to the way I have configured my network? (The service is running in a docker container with port binding, so there might be more in play here than just OPNsense)

The setup seems to work fine for external connections, so I would not think it should be a big security issue having it enabled on LAN only connections.

[pfry]

From your subject I figured you were talking about "Bind states to interface". Personally I'd dig into why normal "State Type: keep state" does not work for you. It suggests a client/server TCP issue, packet loss, and/or partial bypass of the firewall. Packet loss at least should be obvious - it kills performance. I can't think of a practical security exploit for sloppy state offhand, but that doesn't say much.

[muchacha_grande]

Hi FlyveHest, which service is the one needing sloppy states?

I had the same problem using spice proxy from Proxmox.

If I connect to a VM console from a different VLAN through the router, the connection drops after some seconds. The problem is solved using sloppy states, but only for the intended spice proxy port. I made a rule to pass traffic for that port using sloppy states and another to pass the rest of the traffic as normal.

[nero355]

When I connect to a TCP based service on my server from my LAN (port 2 to port 1), I can connect just fine, so the firewall rule work, but after 15 or so seconds, I start seeing "Default deny / state violation rule" in the firewall log, and then the session is disconnected.

If I connect to a VM console from a different VLAN through the router, the connection drops after some seconds.

Sounds like A-Symmetric Routing issues and the "Default DENY / State violation Rule" is always right in my experience :)

So have a good look at the Server you are connecting to and the configuration of it's Interfaces + Routing Rules and let us know what is happening exactly!

(For a quick example see : https://forum.opnsense.org/index.php?topic=53205.msg276222#msg276222)

[muchacha_grande]

Sounds like A-Symmetric Routing issues and the "Default DENY / State violation Rule" is always right in my experience :)

Hi nero355. As the proxmox I'm talking about has 2 interfaces, it could be possible that packets returns from a different interface than the one packets arrived, so it could be possible that it's a case of asymmetric routing.

Thank you for pointing this out.

EDIT:
I've found this, based on nero355s statement: https://forum.proxmox.com/threads/tcp-streams-drop-between-proxmox-vlans-routed-via-virtual-pfsense.134496

[nero355]

Thank you for pointing this out.

It seems like it's time to apply Policy Base Routing on your Server then :)

This is how it looks in my case :

Code Select Expand

# ip rule show
0:      from all lookup local
1:      from 192.168.1.2 lookup 100 proto static
32766:  from all lookup main
32767:  from all lookup default&

Code Select Expand

# ip route show table 100
10.0.0.0/24 via 192.168.1.1 dev eth0 proto static scope siteThis is to avoid the SSH session to time-out when connecting from a host in 10.0.0.0/24 to a SSH Server in 192.168.1.0/24 that has Interfaces in both subnets as shown in my previous reply :)

I have read a lot about this, but in the end all that I needed turned out to be the above! LOL!

[FlyveHest]

From your subject I figured you were talking about "Bind states to interface". Personally I'd dig into why normal "State Type: keep state" does not work for you. It suggests a client/server TCP issue, packet loss, and/or partial bypass of the firewall. Packet loss at least should be obvious - it kills performance. I can't think of a practical security exploit for sloppy state offhand, but that doesn't say much.

Sorry for not getting back, I finalized my setup yesterday evening and during that I have changed some routing issues that I had when I wrote this entry, which I think may have been the cause of the sloppy state issue (packages leaving through one interface but returning through another)

I will check up on it tonight.

[FlyveHest]

Sounds like A-Symmetric Routing issues and the "Default DENY / State violation Rule" is always right in my experience :)

It seems like it's time to apply Policy Base Routing on your Server then :)

Thank you so much, this was the problem, and as I already had a bit of PBR in place it was quick to add a few rules more.

I have tested with a SSH connection that timed out previously, and now works as it should. (Prior I could tcpdump return traffic on the wrong interface, and after the PBR, the traffic was only visible on the correct one)

This has unearthed a new problem, unfortunately, because the issue is still present, but only on services published through Docker.

So, this is probably because there is some form of masquerading or other package mangling happening after routing has happened, leading to the traffic still being sent out the wrong interface.

[nero355]

This has unearthed a new problem, unfortunately, because the issue is still present, but only on services published through Docker.

So, this is probably because there is some form of masquerading or other package mangling happening after routing has happened, leading to the traffic still being sent out the wrong interface.

Do you use MACVLAN for your Docker Network ?

If not : You should! :)

[FlyveHest]

Do you use MACVLAN for your Docker Network ?

If not : You should! :)

I don't no, but the fix was pretty easy, I just had to add another PBR rule for the Docker network bridge and now everything works as it should.

Once again, thanks a bunch for the great help, now that the basic setup is done, I can get into the fun stuff and the main reason I chose to run OPNSense.