Firewall rules are ignored.. but a little bit goes pass...

[Sany]

Hello,

I have a Interface called "VLAN15 (172.16.15.0)" and i have a MQTT Server on a other Subnet (172.16.1.5) i will only allow the MQTT Port, but it's not working.....

Crazy thing, my packets are blocked and pass, but the firewall ignores my rules on VLAN15...

what is todo?

[iMx]

Docker?

State violation, quite possibly the Docker container is replying back on a different IP than the 'destination' that is seen by the firewall.  Or, through different interfaces.

The state entry is created on the initial connection into the interface, source IP -> destination IP, for the destination to be able to send traffic back to the source, it has to match the forward path - same source (destination one way) and destination (source one way), same interfaces being traversed, etc.

... otherwise it is dropped as out of state, a state violation.

[Sany]

Hello,

my opnSense is on a Proxmox Server, the Clients are ESP32..

[iMx]

Click the 'i' next to a drop, what does it show?

[Sany]

Here.. the TCP flags between pass and block are different....

[iMx]

S = SYN
RA = Reset ACK

The SYN is what is needed to be 'seen' to create the state entry on a stateful firewall, to avoid a state violation it needs to see SYN, SYN/ACK and finally ACK (with the source/destination remaining the same, same interfaces traversed, etc).  All of it.  If it only sees part of the connection, it gets dropped as out of state.

If you only see a Reset ACK via the firewall, no prior SYN SYN/ACK - you'd need to look at the logs - it likely means that the 'SYN' or 'SYN/ACK' is not seen via the firewall - asymmetric routing, possibly.

A hack:

- Edit rule
- Enabled Advanced Features
- Under Keep State, None (possibly Sloppy state might work)
- Save/Apply

... but really you should get to the root cause of the problem, why the devices are not sending the initial SYN, SYN/ACK via the firewall... why is the firewall only seeing the Reset ACK.  Asymmetric routing typically.

[Sany]

Thank you, but why my packets randomly blocked and passed ?

[iMx]

Thank you, but why my packets randomly blocked and passed ?

Because they're out of state - A 'SYN' has to be seen before a Reset ACK, see explanation above.  The firewall is not 'seeing' all the packets it should do.

[Sany]

Thank you, the hack worked, but how can i resolv a Asymmetric routing?

My OpnSense are running on a Proxmox Server, there has 3x USB 3.0 Network Cards.
The network cards are connected to the Proxmox Server, and created as virtual "network cards" to the virtual machine with VLAN Aware.

[iMx]

Thank you, the hack worked, but how can i resolv a Asymmetric routing?

My OpnSense are running on a Proxmox Server, there has 3x USB 3.0 Network Cards.
The network cards are connected to the Proxmox Server, and created as virtual "network cards" to the virtual machine with VLAN Aware.

Sounds horrendous ;D

I suspect one of the following is happening:

- The previously 'seen' source and destination is not the same both ways, for example:

a.a.a.a -> b.b.b.b but the reply is c.c.c.c -> a.a.a.a, or b.b.b.b -> d.d.d.d

- You have multiple routes/paths to the destination, not all via the firewall, so the devices can bypass the firewall for the initial 'SYN', so when it only sees tcp flags that should occur later in the connection they're dropped as out of state.

By turning off keep state you're telling the firewall to ignore the various sequences that should occur.  Which is generally not a good thing to do, sometimes necessary but can nearly always be avoided.

[Sany]

Thank you, the hack worked, but how can i resolv a Asymmetric routing?

My OpnSense are running on a Proxmox Server, there has 3x USB 3.0 Network Cards.
The network cards are connected to the Proxmox Server, and created as virtual "network cards" to the virtual machine with VLAN Aware.

Sounds horrendous ;D

I suspect one of the following is happening:

- The previously 'seen' source and destination is not the same both ways, for example:

a.a.a.a -> b.b.b.b but the reply is c.c.c.c -> a.a.a.a, or b.b.b.b -> d.d.d.d

- You have multiple routes/paths to the destination, not all via the firewall, so the devices can bypass the firewall for the initial 'SYN', so when it only sees tcp flags that should occur later in the connection they're dropped as out of state.

By turning off keep state you're telling the firewall to ignore the various sequences that should occur.  Which is generally not a good thing to do, sometimes necessary but can nearly always be avoided.

No i don't have multiple routes to the destination, i have a VLAN15 interface to the 172.16.15.x VLAN with one Route and that is the route from the opnsense.

i tested the VLAN15 to local lan, the VLAN15 rules are empty, when i have multiple routes to the destination that bypasses the firewall, the ping should randomly pass or blocked from the firewall and other way "pass".

But there is nothing after 1500 pings, all are blocked and not bypassed...

[iMx]

You keep saying 'random' - but it's really not random at all. 

The firewall only sees part of the connection, so it drops it.   With Keep state enabled, it needs to see all of the connection.

The other potential option, is that you've got packet loss/drops/weirdness somewhere with your Proxmox/USB NIC setup - opnsense is reacting to what it sees, or doesn't see as the case may be.

[Sany]

You keep saying 'random' - but it's really not random at all. 

The firewall only sees part of the connection, so it drops it.   With Keep state enabled, it needs to see all of the connection.

The other potential option, is that you've got packet loss/drops/weirdness somewhere with your Proxmox/USB NIC setup - opnsense is reacting to what it sees, or doesn't see as the case may be.

Yes random, with my ping test, and block the ping on the VLAN15, there is a packet loss of 100%...  :o

On Proxmox side, i changed the interface for the network cards from virtio to realtek and e1000, without success.
I found a article for this problem and activated 'Bypass firewall rules for traffic on the same interface', without success...

But then i was suprised, i started "tracert 172.16.1.5" from my Laptop in VLAN15....
And the Only Route to 172.16.1.5 is directly "172.16.1.1", but the network card has the gateway "172.16.15.1".

doesn't my trace have to go the route "172.16.15.36 -> 172.16.15.1 -> 172.16.1.1" ?