[SOLVED] Replies being dropped with "Default deny / state violation rule"

[Isabella Borgward]

Scenario: on TNET interface we have a L3 routed network, one example network is 10.10.45.0/24. Routes to these networks are configured in System: Routes: Configuration.

Outbound NAT policies for these networks have been created manually, I assume they're getting hits because at the far end I can see my test traffic originating from the expected IP.

Here's the problem - the replies are being dropped because of "Default deny / state violation rule". My guess here is that the firewall knows the matching state for these rules, otherwise it wouldn't know the private IP they're for. So what is the real issue?
Devices in the TNET subnet work fine, it's just the routed subnets that don't work.

Code Select Expand

TNET        2025-06-17T08:42:25    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:17    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:09    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:05    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:01    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:59    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:57    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:56    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:55    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:54    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
WAN        2025-06-17T08:41:54    45.76.130.220:14043    91.2.119.28:80    tcp    let out anything from firewall host itself (force gw)   

[Patrick M. Hausen]

Is there possibly a case of asymmetric routing at work? I.e. do packets pass through OPNsense in one direction but take a different route in the reverse one?

[Isabella Borgward]

No, there is a single WAN gateway and a single gateway to reach these routed subnets on TNET.

[pfry]

[...]My guess here is that the firewall knows the matching state for these rules, otherwise it wouldn't know the private IP they're for.[...]

Reasonable. Hit the "i" on the right for one of those live logs and see if there's anything unusual (e.g. TCP flags), and look at "Firewall: Diagnostics: States" to make certain the bidirectional state is set up properly. I've seen pf drop TCP sessions that violate the handshaking rules, and they log as default denies (in my ruleset).

[Isabella Borgward]

tcpflags on the dropped packets are 'SA'.
ICMP ping has the same issue.
If I click on the rule ID in the info popup, it opens a tab which then immediately closes itself.

The state table looks "reasonable" to me - the only oddity is that the matching rule is alleged to be "let anything out of the firewall itself", which this traffic is not. But I compared it to another firewall and that's the same matching rule name.

[pfry]

tcpflags on the dropped packets are 'SA'.

From my firewall, I would expect "S". Yours looks kinda like a new state setting up halfway into the process. I haven't seen this in my own setup. You wouldn't happen to have synproxy enabled on the pass rule for these sessions? It's an option under "State Type" near the bottom of the "Advanced features" in the rule definition. I can't see this affecting ICMP, and synproxy should only break under certain conditions.

[...] If I click on the rule ID in the info popup, it opens a tab which then immediately closes itself.

I get that if the linked rule is automatic, I believe. I mask the automatic default block with my own, so I only see logs for the former when a packet somehow escapes my rules.

[...] the only oddity is that the matching rule is alleged to be "let anything out of the firewall itself" [...]

Yep, that's normal.

[Isabella Borgward]

You wouldn't happen to have synproxy enabled on the pass rule for these sessions?

I have tried every option under state type on this rule, none of them work.

[Isabella Borgward]

OK.....not the fault of the edge Opnsense firewall...the issue was the downstream L3 router, just routing the replies coming back from the internet, back out to the edge firewall again. Once I told it to just route things to the correct places, it's now working.
No wonder OpnSense and I were very confused!