Hi,
after upgrading from
26.1.x to
26.7.1, I noticed something odd that I haven't seen before.
Immediately after the upgrade, my firewall started logging a large number of state-insert,block events.
The reason I'm posting this is the timing. I've been collecting all OPNsense firewall logs in Graylog for quite some time, so I searched the last 30 days.
Result:
- Before the upgrade: 0 occurrences of state-insert
- Immediately after upgrading to 26.7.1: thousands of state-insert,block events
At first I assumed this was a WebServer issue because that was the first service that stopped working. However, nginx and PHP-FPM were both running normally and TLS handshakes completed successfully, so I started investigating the firewall instead.
The firewall log shows entries like these:
match,pass,in,...
192.168.1.2 -> 192.168.243.10:12345
state-insert,block,in,...
192.168.1.2 -> 192.168.243.10:12345
The corresponding PF rule is:
@218 pass in log quick on igc0_vlan4096 inet
flags S/SA keep state
label "88f5cbb7-6e4c-47da-9811-ca92e09741a7"
divert-to 8000
Evaluations: 92502
Packets: 23425
States: 7
State Creations: 1093
For reference, this is the current output of pfctl -si:
State Table
current entries: 5717
Counters
state-insert: 7184
state-limit: 0
memory: 0
The state table is clearly not full, so this doesn't appear to be a state table limit issue.
My setup:
- OPNsense 26.7.1 (upgraded from 26.1.x)
- Suricata Inline IPS
- Hairpin NAT / NAT Reflection for exposed https service which failed after the upgrade
- Graylog for centralized firewall logging
I'm not claiming that Suricata or NAT Reflection are the cause. They are simply part of my environment.
The only thing I can say with confidence is :
- 30 days of Graylog history before the upgrade: 0 state-insert events.
- Immediately after upgrading to 26.7.1: thousands of state-insert,block events.
I'll also attach a screenshot from Graylog showing exactly that.
Has anyone else seen this after upgrading to 26.7.x?
Could this be related to the FreeBSD 15.1 / PF changes?
I did a rollback to a created snapshot, as I had not much time to debug.
Cheers,
fb
I have now found another example which does not involve hairpin access to an internal server.
vlan0.695,state-insert,block,in,4,0x0,,255,41990,0,none,17,udp,76,10.1.1.14,10.1.1.1,49753,123,56
10.1.1.1 is the firewall address on this VLAN.
There is a destination NAT rule which redirects outbound UDP port 123 traffic to the firewall itself, so clients are forced to use the local NTP service. The interface firewall rule allows the subnet to access the firewall address on UDP ports 53 and 123.
This means the issue is not limited to hairpin NAT towards an internal server. It also occurs with UDP traffic destination-NATed to a service running locally on the firewall.