26.7.1: "state-insert,block" events started immediately after upgrade

Started by fastboot, July 24, 2026, 08:02:56 AM

Previous topic - Next topic
Hi,

after upgrading from 26.1.x to 26.7.1, I noticed something odd that I haven't seen before.

Immediately after the upgrade, my firewall started logging a large number of state-insert,block events.

The reason I'm posting this is the timing. I've been collecting all OPNsense firewall logs in Graylog for quite some time, so I searched the last 30 days.

Result:

  • Before the upgrade: 0 occurrences of state-insert
  • Immediately after upgrading to 26.7.1: thousands of state-insert,block events

At first I assumed this was a WebServer issue because that was the first service that stopped working. However, nginx and PHP-FPM were both running normally and TLS handshakes completed successfully, so I started investigating the firewall instead.

The firewall log shows entries like these:

match,pass,in,...
192.168.1.2 -> 192.168.243.10:12345

state-insert,block,in,...
192.168.1.2 -> 192.168.243.10:12345

The corresponding PF rule is:

@218 pass in log quick on igc0_vlan4096 inet
flags S/SA keep state
label "88f5cbb7-6e4c-47da-9811-ca92e09741a7"
divert-to 8000

Evaluations:      92502
Packets:          23425
States:               7
State Creations:   1093

For reference, this is the current output of pfctl -si:

State Table
  current entries: 5717

Counters
  state-insert: 7184
  state-limit: 0
  memory: 0

The state table is clearly not full, so this doesn't appear to be a state table limit issue.

My setup:

  • OPNsense 26.7.1 (upgraded from 26.1.x)
  • Suricata Inline IPS
  • Hairpin NAT / NAT Reflection for exposed https service which failed after the upgrade
  • Graylog for centralized firewall logging

I'm not claiming that Suricata or NAT Reflection are the cause. They are simply part of my environment.

The only thing I can say with confidence is :

  • 30 days of Graylog history before the upgrade: 0 state-insert events.
  • Immediately after upgrading to 26.7.1: thousands of state-insert,block events.

I'll also attach a screenshot from Graylog showing exactly that.

Has anyone else seen this after upgrading to 26.7.x?

Could this be related to the FreeBSD 15.1 / PF changes?

I did a rollback to a created snapshot, as I had not much time to debug.


Cheers,

fb



I have now found another example which does not involve hairpin access to an internal server.

vlan0.695,state-insert,block,in,4,0x0,,255,41990,0,none,17,udp,76,10.1.1.14,10.1.1.1,49753,123,56

10.1.1.1 is the firewall address on this VLAN.

There is a destination NAT rule which redirects outbound UDP port 123 traffic to the firewall itself, so clients are forced to use the local NTP service. The interface firewall rule allows the subnet to access the firewall address on UDP ports 53 and 123.

This means the issue is not limited to hairpin NAT towards an internal server. It also occurs with UDP traffic destination-NATed to a service running locally on the firewall.

I am seeing this too in relation to Suricata in Divert mode. Inbound packets on my WAN rules are getting evaluated twice (occasionally 3 times), first time is a pass and then the second (or last) time it is dropped for "state-insert,block". If I remove the divert-to from the rule, then it works normally.

Example:

root@home:~ # tail -f /var/log/filter/latest.log | grep 5060
<134>1 2026-07-25T20:52:58-04:00 home.example.com filterlog 30391 - [meta sequenceId="420777"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,22381,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:52:58-04:00 home.example.com filterlog 30391 - [meta sequenceId="420778"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,22381,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:52:58-04:00 home.example.com filterlog 30391 - [meta sequenceId="420779"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,22381,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:52:59-04:00 home.example.com filterlog 30391 - [meta sequenceId="420785"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,22403,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:52:59-04:00 home.example.com filterlog 30391 - [meta sequenceId="420786"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,22403,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:52:59-04:00 home.example.com filterlog 30391 - [meta sequenceId="420795"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,22456,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:52:59-04:00 home.example.com filterlog 30391 - [meta sequenceId="420796"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,22456,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:53:00-04:00 home.example.com filterlog 30391 - [meta sequenceId="420799"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,40213,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:00-04:00 home.example.com filterlog 30391 - [meta sequenceId="420800"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,40213,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:00-04:00 home.example.com filterlog 30391 - [meta sequenceId="420801"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,40213,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:00-04:00 home.example.com filterlog 30391 - [meta sequenceId="420806"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,40233,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:00-04:00 home.example.com filterlog 30391 - [meta sequenceId="420807"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,40233,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:01-04:00 home.example.com filterlog 30391 - [meta sequenceId="420811"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,40250,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:01-04:00 home.example.com filterlog 30391 - [meta sequenceId="420812"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,40250,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:01-04:00 home.example.com filterlog 30391 - [meta sequenceId="420813"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,40271,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:01-04:00 home.example.com filterlog 30391 - [meta sequenceId="420814"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,40271,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:02-04:00 home.example.com filterlog 30391 - [meta sequenceId="420818"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,40298,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:02-04:00 home.example.com filterlog 30391 - [meta sequenceId="420819"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,40298,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:02-04:00 home.example.com filterlog 30391 - [meta sequenceId="420825"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,40337,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:02-04:00 home.example.com filterlog 30391 - [meta sequenceId="420826"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,40337,0,none,17,udp,1190,34.210.91.112,10.7.0.10,5060,5060,1170
<134>1 2026-07-25T20:53:03-04:00 home.example.com filterlog 30391 - [meta sequenceId="420830"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,22595,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:53:03-04:00 home.example.com filterlog 30391 - [meta sequenceId="420831"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,22595,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:53:03-04:00 home.example.com filterlog 30391 - [meta sequenceId="420834"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,22606,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:53:03-04:00 home.example.com filterlog 30391 - [meta sequenceId="420835"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,22606,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:53:04-04:00 home.example.com filterlog 30391 - [meta sequenceId="420838"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,match,pass,in,4,0x68,,122,22636,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168
<134>1 2026-07-25T20:53:04-04:00 home.example.com filterlog 30391 - [meta sequenceId="420839"] 187,,,55e761c8-fbf4-4921-a72a-5824b67416b7,igc0,state-insert,block,in,4,0x68,,122,22636,0,none,17,udp,1188,34.226.36.34,10.7.0.10,5060,5060,1168


The affected rules are associated with destination NATs. In the example its TCP/UDP 5060 (SIP) from the WAN to the DMZ server 10.7.0.10, but I also have rules for 80 and 443 with the same results.

Edit: I forgot to mention that mine was working fine on 26.1, and I only saw this behavior after updating to 26.7.1.

Hi @SonicJoe,


that was my first assumption. I disabled the divert to of the rules that allow the flows. But in my case it did not help. I also checked the corresponding logs for suricata and could not find any blocks. For example "dest_port:123" should trigger a hit in the suricata logs, as this was blocked.

@franco: Help please



Quote from: fastboot on Today at 08:00:15 AMHi @SonicJoe,


that was my first assumption. I disabled the divert to of the rules that allow the flows. But in my case it did not help. I also checked the corresponding logs for suricata and could not find any blocks. For example "dest_port:123" should trigger a hit in the suricata logs, as this was blocked.

@franco: Help please



I should clarify that I have other rules with divert-to set that work fine. It is only the rules on the WAN interface. I suspect something about the Destination NAT rules is causing confusion when the packet comes back from Suricata and gets re-inserted in the flow. I agree that Suricata is not blocking anything. I did see evidence that Suricata was evaluating the packets though:

app_layer.flow.sip_udp                                      | Total                    | 10
app_layer.tx.sip_udp                                          | Total                    | 57