Floating firewall rules don't work (fully) with IPsec policy based VPN?

Started by onnieoneone, February 22, 2026, 09:42:55 AM

Previous topic - Next topic
Print
Go Down Pages1
User actions
February 22, 2026, 09:42:55 AM Last Edit: February 22, 2026, 09:45:29 AM by onnieoneone
Did I find an OPNSense bug?

Or, more likely am I misunderstanding something subtle about how pf and IPsec work together?

I'll detail below in all its gory detail how I think pf is not tracking connections correctly when using IPsec policy based connections.

Here goes.

First off, I think I have set up the IPSec side correctly for a site-to-site VPN between my OPNsense (bang up to date 26.1.2) router at site1 (IPv4 WAN address behind CPE NAT) and an old Unifi Security Gateway 3 (basically Vyatta) at site2 (also IPv4 behind NAT).

I see the Child SAs looking good and even ICMP traffic passing from either side to the other and back.

The subnets in question I am tunnelling at the moment from site1 are 10.1.8.0/24 and 10.2.4.0/24 (and a few others) at site2, that is, all 10.1.X.Y hosts are at site1 and 10.2.X.Y hosts are at site2.

I have a Floating firewall ICMP allow rule that creates this pf rule on the OPNsense at site1:

Code Select
site1-opnsense # grep 'Allow RFC' /tmp/rules.debug
pass log quick inet proto icmp from $rfc1918_ipv4_nets to $rfc1918_ipv4_nets keep state label "f8d0246d9d3d9c9afd3f531459d75811" # Allow RFC1918 networks ICMP access
site1-opnsense # pfctl -t rfc1918_ipv4_nets -T show
   10.0.0.0/8
   172.16.0.0/12
   192.168.0.0/16

When I try to ping between a couple of hosts (10.1.8.27 at site1 and 10.2.4.43 at site2) everything looks good.

Firstly, a ping from site1 to site2:

Code Select
site1-host # ping -c 1 -I 10.1.8.27 10.2.4.43
PING 10.2.4.43 (10.2.4.43) from 10.1.8.27 : 56(84) bytes of data.
64 bytes from 10.2.4.43: icmp_seq=1 ttl=62 time=296 ms

--- 10.2.4.43 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 296.325/296.325/296.325/0.000 ms

site1-opnsense # tail -f /var/log/filter/latest.log|grep 10.2.4.43
<134>1 2026-02-21T17:19:51+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="50416"] 214,,,f8d0246d9d3d9c9afd3f531459d75811,lagg0_vlan1018,match,pass,in,4,0x0,,64,30186,0,DF,1,icmp,84,10.1.8.27,10.2.4.43,datalength=64
<134>1 2026-02-21T17:19:51+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="50417"] 214,,,f8d0246d9d3d9c9afd3f531459d75811,igb3,match,pass,in,4,0x0,,63,24807,0,none,1,icmp,84,10.2.4.43,10.1.8.27,datalength=64


site1-opnsense # tcpdump -i enc0 -n -vvvv -tttt host 10.2.4.43
tcpdump: listening on enc0, link-type ENC (OpenBSD encapsulated IP), snapshot length 262144 bytes
2026-02-21 17:19:50.447230 (authentic,confidential): SPI 0xc90a3520: IP (tos 0x0, ttl 63, id 30186, offset 0, flags [DF], proto ICMP (1), length 84, bad cksum a476 (->a576)!)
    10.1.8.27 > 10.2.4.43: ICMP echo request, id 52438, seq 1, length 64
2026-02-21 17:19:50.742370 (authentic,confidential): SPI 0xc233a0d7: IP (tos 0x0, ttl 63, id 24807, offset 0, flags [none], proto ICMP (1), length 84)
    10.2.4.43 > 10.1.8.27: ICMP echo reply, id 52438, seq 1, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

And again a trial ping from site2 to site1:

Code Select
site2-host # ping -c 1 -I 10.2.4.43 10.1.8.27
PING 10.1.8.27 (10.1.8.27) from 10.2.4.43 : 56(84) bytes of data.
64 bytes from 10.1.8.27: icmp_seq=1 ttl=62 time=298 ms

--- 10.1.8.27 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 298.207/298.207/298.207/0.000 ms

site1-opnsense # tail -f /var/log/filter/latest.log|grep 10.2.4.43
<134>1 2026-02-21T17:21:54+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="50646"] 214,,,f8d0246d9d3d9c9afd3f531459d75811,igb3,match,pass,in,4,0x0,,63,54800,0,DF,1,icmp,84,10.2.4.43,10.1.8.27,datalength=64
<134>1 2026-02-21T17:21:54+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="50647"] 214,,,f8d0246d9d3d9c9afd3f531459d75811,lagg0_vlan1018,match,pass,out,4,0x0,,62,54800,0,DF,1,icmp,84,10.2.4.43,10.1.8.27,datalength=64

site1-opnsense # tcpdump -i enc0 -n -vvvv -tttt host 10.2.4.43
tcpdump: listening on enc0, link-type ENC (OpenBSD encapsulated IP), snapshot length 262144 bytes
2026-02-21 17:21:53.976534 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 54800, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.4.43 > 10.1.8.27: ICMP echo request, id 10057, seq 1, length 64
2026-02-21 17:21:53.977528 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x0, ttl 63, id 43093, offset 0, flags [none], proto ICMP (1), length 84, bad cksum b20b (->b30b)!)
    10.1.8.27 > 10.2.4.43: ICMP echo reply, id 10057, seq 1, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

So everything looks great, passing traffic, firewall is working. This is even working for my other child SA networks (10.2.1.0/24, 10.2.2.0/24, etc).


The problem then comes in with some other traffic. I try to SSH between sites and it's just not working.

Firstly my SAs:
Code Select
site1-opnsense # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
ca24ede1-a222-4a6e-91dc-2c5720143fe6: #4, ESTABLISHED, <IKE-STUFF>
  local  '<LOCAL-UNNATTED-WAN-IP>' @ <LOCAL-NATTED-WAN-IP>[4500]
  remote '<REMOTE-NATTED-WAN-IP>' @ <REMOTE-UNNATTED-WAN-IP>[4500]
  <CRYPTO-CIPHERS-AND-STUFF>
  established 8811s ago, rekeying in 4253s
  77828303-0d3c-4568-8a06-7093234cd740: #15, reqid 1, INSTALLED, TUNNEL-in-UDP, ESP:<CRYPTO-CIPHERS-AND-STUFF>
    installed 731s ago, rekeying in 2610s, expires in 3230s
    in  c4e9cb8f,   1008 bytes,    12 packets,    55s ago
    out cbb6212d,   1824 bytes,    12 packets,    55s ago
    local  10.1.8.0/24
    remote 10.2.2.0/24
ca24ede1-a222-4a6e-91dc-2c5720143fe6: #6, ESTABLISHED, <IKE-STUFF>
  local  '<LOCAL-UNNATTED-WAN-IP>' @ <LOCAL-NATTED-WAN-IP>[4500]
  remote '<REMOTE-NATTED-WAN-IP>' @ <REMOTE-UNNATTED-WAN-IP>[4500]
  <CRYPTO-CIPHERS-AND-STUFF>
  established 8811s ago, rekeying in 5563s
  77828303-0d3c-4568-8a06-7093234cd740: #14, reqid 3, INSTALLED, TUNNEL-in-UDP, ESP:<CRYPTO-CIPHERS-AND-STUFF>
    installed 819s ago, rekeying in 2586s, expires in 3142s
    in  c68d83fa,   1092 bytes,    13 packets,    55s ago
    out c9f3d367,   1976 bytes,    13 packets,    56s ago
    local  10.1.8.0/24
    remote 10.2.0.0/24
  77828303-0d3c-4568-8a06-7093234cd740: #16, reqid 4, INSTALLED, TUNNEL-in-UDP, ESP:<CRYPTO-CIPHERS-AND-STUFF>
    installed 248s ago, rekeying in 3023s, expires in 3713s
    in  cfadb11d,    336 bytes,     4 packets,    55s ago
    out c0a8bbd4,    608 bytes,     4 packets,    56s ago
    local  10.1.8.0/24
    remote 10.2.8.0/24
ca24ede1-a222-4a6e-91dc-2c5720143fe6: #5, ESTABLISHED, <IKE-STUFF>
  local  '<LOCAL-UNNATTED-WAN-IP>' @ <LOCAL-NATTED-WAN-IP>[4500]
  remote '<REMOTE-NATTED-WAN-IP>' @ <REMOTE-UNNATTED-WAN-IP>[4500]
  <CRYPTO-CIPHERS-AND-STUFF>
  established 8811s ago, rekeying in 4473s
  77828303-0d3c-4568-8a06-7093234cd740: #13, reqid 2, INSTALLED, TUNNEL-in-UDP, ESP:<CRYPTO-CIPHERS-AND-STUFF>
    installed 830s ago, rekeying in 2457s, expires in 3131s
    in  c4bad24d,   3525 bytes,    25 packets,    55s ago
    out ce6bb79c,   4384 bytes,    22 packets,    55s ago
    local  10.1.8.0/24
    remote 10.2.4.0/24

So my SPIs for this are c4bad24d and ce6bb79c and furthermore in the packet captures it looks like the right traffic is hitting the right SPI.

I created some test firewall rules, one right below the other in the (old) UI and here's the pf result:

Code Select
site1-opnsense # grep TEST /tmp/rules.debug
pass log quick inet proto tcp from $temp_10_1_8_27 to $temp_10_2_4_43 port {22} keep state label "04dcefb83dcccd488cdddb48c535eab9" # TEST IPSec site1 to site2 SSH access
pass log quick inet proto tcp from $temp_10_2_4_43 to $temp_10_1_8_27 port {22} keep state label "fc037937eb7b02d7efbd5244c7b9c70f" # TEST IPSec site2 to site1 SSH access

site1-opnsense # pfctl -t temp_10_1_8_27 -T show
   10.1.8.27
site1-opnsense # pfctl -t temp_10_2_4_43 -T show
   10.2.4.43

So here's a successful SSH from site2 to site1 (I just ctrl+c when it asks for fingerprint validation):

Code Select
site2-host # ssh 10.1.8.27
The authenticity of host '10.1.8.27 (10.1.8.27)' can't be established.
ED25519 key fingerprint is SHA256:Zwag/SC/O5MPZm8A22/63FEN4YB+4c/zj/wFLs2cZrE.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? ^C

site1-opnsense # tail -f /var/log/filter/latest.log|grep 10.2.4.43
<134>1 2026-02-21T17:25:32+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="50972"] 207,,,fc037937eb7b02d7efbd5244c7b9c70f,igb3,match,pass,in,4,0x10,,63,59405,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,33100,22,0,S,3196047947,,64240,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:25:32+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="50973"] 207,,,fc037937eb7b02d7efbd5244c7b9c70f,lagg0_vlan1018,match,pass,out,4,0x10,,62,59405,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,33100,22,0,S,3196047947,,64240,,mss;sackOK;TS;nop;wscale
^C

site1-opnsense # tcpdump -i enc0 -n -vvvv -tttt host 10.2.4.43
tcpdump: listening on enc0, link-type ENC (OpenBSD encapsulated IP), snapshot length 262144 bytes
2026-02-21 17:25:31.544381 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59405, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [S], cksum 0xc0d8 (correct), seq 3196047947, win 64240, options [mss 1460,sackOK,TS val 274155881 ecr 0,nop,wscale 7], length 0
2026-02-21 17:25:31.545438 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 1a74 (->1b74)!)
    10.1.8.27.22 > 10.2.4.43.33100: Flags [S.], cksum 0xf4ab (correct), seq 4163488344, ack 3196047948, win 65160, options [mss 1460,sackOK,TS val 2825870737 ecr 274155881,nop,wscale 7], length 0                                                                                       
2026-02-21 17:25:31.837392 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59406, offset 0, flags [DF], proto TCP (6), length 52)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [.], cksum 0x1ee6 (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 274156174 ecr 2825870737], length 0
2026-02-21 17:25:31.843063 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59407, offset 0, flags [DF], proto TCP (6), length 85)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [P.], cksum 0x627a (correct), seq 1:34, ack 1, win 502, options [nop,nop,TS val 274156174 ecr 2825870737], length 33: SSH: SSH-2.0-OpenSSH_10.0p2 Debian-7
2026-02-21 17:25:31.843428 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x0, ttl 63, id 34597, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 9356 (->9456)!)
    10.1.8.27.22 > 10.2.4.43.33100: Flags [.], cksum 0x1d94 (correct), seq 1, ack 34, win 509, options [nop,nop,TS val 2825871035 ecr 274156174], length 0
2026-02-21 17:25:31.848758 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x0, ttl 63, id 34598, offset 0, flags [DF], proto TCP (6), length 85, bad cksum 9334 (->9434)!)
    10.1.8.27.22 > 10.2.4.43.33100: Flags [P.], cksum 0x6123 (correct), seq 1:34, ack 34, win 509, options [nop,nop,TS val 2825871040 ecr 274156174], length 33: SSH: SSH-2.0-OpenSSH_10.0p2 Debian-7
2026-02-21 17:25:32.141973 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59408, offset 0, flags [DF], proto TCP (6), length 52)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [.], cksum 0x1c45 (correct), seq 34, ack 34, win 502, options [nop,nop,TS val 274156478 ecr 2825871040], length 0
2026-02-21 17:25:32.142833 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x0, ttl 63, id 34599, offset 0, flags [DF], proto TCP (6), length 796, bad cksum 906c (->916c)!)
    10.1.8.27.22 > 10.2.4.43.33100: Flags [P.], cksum 0x1607 (correct), seq 34:778, ack 34, win 509, options [nop,nop,TS val 2825871334 ecr 274156478], length 744                                                                                                                       
2026-02-21 17:25:32.148146 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59410, offset 0, flags [DF], proto TCP (6), length 172)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [P.], cksum 0x502c (correct), seq 1482:1602, ack 34, win 502, options [nop,nop,TS val 274156479 ecr 2825871040], length 120
2026-02-21 17:25:32.148208 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59411, offset 0, flags [DF], proto TCP (6), length 1422)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [.], cksum 0x766c (correct), seq 34:1404, ack 34, win 502, options [nop,nop,TS val 274156479 ecr 2825871040], length 1370
2026-02-21 17:25:32.148227 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59412, offset 0, flags [DF], proto TCP (6), length 250)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [P.], cksum 0xe306 (correct), seq 1404:1602, ack 34, win 502, options [nop,nop,TS val 274156479 ecr 2825871040], length 198
2026-02-21 17:25:32.149161 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x0, ttl 63, id 34600, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 9347 (->9447)!)
    10.1.8.27.22 > 10.2.4.43.33100: Flags [.], cksum 0xb46e (correct), seq 778, ack 34, win 509, options [nop,nop,TS val 2825871341 ecr 274156478,nop,nop,sack 1 {1482:1602}], length 0
2026-02-21 17:25:32.149244 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x0, ttl 63, id 34601, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 9346 (->9446)!)
    10.1.8.27.22 > 10.2.4.43.33100: Flags [.], cksum 0xae59 (correct), seq 778, ack 1602, win 497, options [nop,nop,TS val 2825871341 ecr 274156479,nop,nop,sack 1 {1482:1602}], length 0
2026-02-21 17:25:32.441484 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59413, offset 0, flags [DF], proto TCP (6), length 100)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [P.], cksum 0xdadc (correct), seq 1602:1650, ack 778, win 497, options [nop,nop,TS val 274156778 ecr 2825871334], length 48
2026-02-21 17:25:32.449304 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x0, ttl 63, id 34602, offset 0, flags [DF], proto TCP (6), length 544, bad cksum 9165 (->9265)!)
    10.1.8.27.22 > 10.2.4.43.33100: Flags [P.], cksum 0x6d4c (correct), seq 778:1270, ack 1650, win 497, options [nop,nop,TS val 2825871641 ecr 274156778], length 492
2026-02-21 17:25:32.782963 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59414, offset 0, flags [DF], proto TCP (6), length 52)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [.], cksum 0x0c4f (correct), seq 1650, ack 1270, win 494, options [nop,nop,TS val 274157119 ecr 2825871641], length 0
2026-02-21 17:25:35.391466 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59415, offset 0, flags [DF], proto TCP (6), length 52)                                                                                                                                     
    10.2.4.43.33100 > 10.1.8.27.22: Flags [F.], cksum 0x021d (correct), seq 1650, ack 1270, win 494, options [nop,nop,TS val 274159728 ecr 2825871641], length 0
2026-02-21 17:25:35.394564 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x0, ttl 63, id 34603, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 9350 (->9450)!)
    10.1.8.27.22 > 10.2.4.43.33100: Flags [F.], cksum 0xf697 (correct), seq 1270, ack 1651, win 497, options [nop,nop,TS val 2825874586 ecr 274159728], length 0
2026-02-21 17:25:35.688799 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x10, ttl 63, id 59416, offset 0, flags [DF], proto TCP (6), length 52)
    10.2.4.43.33100 > 10.1.8.27.22: Flags [.], cksum 0xf573 (correct), seq 1651, ack 1271, win 494, options [nop,nop,TS val 274160023 ecr 2825874586], length 0
^C                                                                                                                                           
19 packets captured                                                                                                                         
19 packets received by filter                             
0 packets dropped by kernel 

Now the flow that doesn't work, site1 to site2 SSH:

Code Select
site1-host # ssh 10.2.4.43
ssh: connect to host 10.2.4.43 port 22: Connection timed out

site1-opnsense # tail -f /var/log/filter/latest.log|grep 10.2.4.43
<134>1 2026-02-21T17:44:03+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52450"] 206,,,04dcefb83dcccd488cdddb48c535eab9,lagg0_vlan1018,match,pass,in,4,0x10,,64,18105,0,DF,6,tcp,60,10.1.8.27,10.2.4.43,36596,22,0,S,2269409707,,64240,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:44:03+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52451"] 24,,,02f4bab031b57d1e30553ce08e0ec131,igb3,match,block,in,4,0x0,,63,0,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,22,36596,0,SA,3655318959,2269409708,65160,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:44:05+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52458"] 24,,,02f4bab031b57d1e30553ce08e0ec131,igb3,match,block,in,4,0x0,,63,0,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,22,36596,0,SA,3655318959,2269409708,65160,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:44:06+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52460"] 24,,,02f4bab031b57d1e30553ce08e0ec131,igb3,match,block,in,4,0x0,,63,0,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,22,36596,0,SA,3655318959,2269409708,65160,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:44:07+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52462"] 24,,,02f4bab031b57d1e30553ce08e0ec131,igb3,match,block,in,4,0x0,,63,0,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,22,36596,0,SA,3655318959,2269409708,65160,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:44:08+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52464"] 24,,,02f4bab031b57d1e30553ce08e0ec131,igb3,match,block,in,4,0x0,,63,0,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,22,36596,0,SA,3655318959,2269409708,65160,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:44:10+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52466"] 24,,,02f4bab031b57d1e30553ce08e0ec131,igb3,match,block,in,4,0x0,,63,0,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,22,36596,0,SA,3655318959,2269409708,65160,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:44:15+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52472"] 24,,,02f4bab031b57d1e30553ce08e0ec131,igb3,match,block,in,4,0x0,,63,0,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,22,36596,0,SA,3655318959,2269409708,65160,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:44:23+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52480"] 24,,,02f4bab031b57d1e30553ce08e0ec131,igb3,match,block,in,4,0x0,,63,0,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,22,36596,0,SA,3655318959,2269409708,65160,,mss;sackOK;TS;nop;wscale
<134>1 2026-02-21T17:44:39+01:00 site1-opnsense filterlog 44466 - [meta sequenceId="52500"] 24,,,02f4bab031b57d1e30553ce08e0ec131,igb3,match,block,in,4,0x0,,63,0,0,DF,6,tcp,60,10.2.4.43,10.1.8.27,22,36596,0,SA,3655318959,2269409708,65160,,mss;sackOK;TS;nop;wscale


site1-opnsense # tcpdump -i enc0 -n -vvvv -tttt host 10.2.4.43
tcpdump: listening on enc0, link-type ENC (OpenBSD encapsulated IP), snapshot length 262144 bytes
2026-02-21 17:44:03.173792 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x10, ttl 63, id 18105, offset 0, flags [DF], proto TCP (6), length 60, bad cksum d3aa (->d4aa)!)
    10.1.8.27.36596 > 10.2.4.43.22: Flags [S], cksum 0xa86e (correct), seq 2269409707, win 64240, options [mss 1460,sackOK,TS val 2826982365 ecr 0,nop,wscale 7], length 0
2026-02-21 17:44:03.468275 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.22 > 10.1.8.27.36596: Flags [S.], cksum 0xbbf2 (correct), seq 3655318959, ack 2269409708, win 65160, options [mss 1460,sackOK,TS val 275267803 ecr 2826982365,nop,wscale 7], length 0
2026-02-21 17:44:04.200440 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x10, ttl 63, id 18106, offset 0, flags [DF], proto TCP (6), length 60, bad cksum d3a9 (->d4a9)!)
    10.1.8.27.36596 > 10.2.4.43.22: Flags [S], cksum 0xa46b (correct), seq 2269409707, win 64240, options [mss 1460,sackOK,TS val 2826983392 ecr 0,nop,wscale 7], length 0
2026-02-21 17:44:05.224458 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x10, ttl 63, id 18107, offset 0, flags [DF], proto TCP (6), length 60, bad cksum d3a8 (->d4a8)!)
    10.1.8.27.36596 > 10.2.4.43.22: Flags [S], cksum 0xa06b (correct), seq 2269409707, win 64240, options [mss 1460,sackOK,TS val 2826984416 ecr 0,nop,wscale 7], length 0
2026-02-21 17:44:05.521403 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.22 > 10.1.8.27.36596: Flags [S.], cksum 0xb3f0 (correct), seq 3655318959, ack 2269409708, win 65160, options [mss 1460,sackOK,TS val 275269853 ecr 2826982365,nop,wscale 7], length 0
2026-02-21 17:44:06.248612 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x10, ttl 63, id 18108, offset 0, flags [DF], proto TCP (6), length 60, bad cksum d3a7 (->d4a7)!)
    10.1.8.27.36596 > 10.2.4.43.22: Flags [S], cksum 0x9c6b (correct), seq 2269409707, win 64240, options [mss 1460,sackOK,TS val 2826985440 ecr 0,nop,wscale 7], length 0
2026-02-21 17:44:06.539606 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.22 > 10.1.8.27.36596: Flags [S.], cksum 0xaff0 (correct), seq 3655318959, ack 2269409708, win 65160, options [mss 1460,sackOK,TS val 275270877 ecr 2826982365,nop,wscale 7], length 0
2026-02-21 17:44:07.276362 (authentic,confidential): SPI 0xce6bb79c: IP (tos 0x10, ttl 63, id 18109, offset 0, flags [DF], proto TCP (6), length 60, bad cksum d3a6 (->d4a6)!)
    10.1.8.27.36596 > 10.2.4.43.22: Flags [S], cksum 0x9867 (correct), seq 2269409707, win 64240, options [mss 1460,sackOK,TS val 2826986468 ecr 0,nop,wscale 7], length 0
2026-02-21 17:44:07.567066 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.22 > 10.1.8.27.36596: Flags [S.], cksum 0xabec (correct), seq 3655318959, ack 2269409708, win 65160, options [mss 1460,sackOK,TS val 275271905 ecr 2826982365,nop,wscale 7], length 0
2026-02-21 17:44:08.594028 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.22 > 10.1.8.27.36596: Flags [S.], cksum 0xa7ea (correct), seq 3655318959, ack 2269409708, win 65160, options [mss 1460,sackOK,TS val 275272931 ecr 2826982365,nop,wscale 7], length 0
2026-02-21 17:44:10.609542 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.22 > 10.1.8.27.36596: Flags [S.], cksum 0xa00a (correct), seq 3655318959, ack 2269409708, win 65160, options [mss 1460,sackOK,TS val 275274947 ecr 2826982365,nop,wscale 7], length 0
2026-02-21 17:44:14.801735 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.22 > 10.1.8.27.36596: Flags [S.], cksum 0x8faa (correct), seq 3655318959, ack 2269409708, win 65160, options [mss 1460,sackOK,TS val 275279139 ecr 2826982365,nop,wscale 7], length 0
2026-02-21 17:44:22.994395 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.22 > 10.1.8.27.36596: Flags [S.], cksum 0x6faa (correct), seq 3655318959, ack 2269409708, win 65160, options [mss 1460,sackOK,TS val 275287331 ecr 2826982365,nop,wscale 7], length 0
2026-02-21 17:44:39.122500 (authentic,confidential): SPI 0xc4bad24d: IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.4.43.22 > 10.1.8.27.36596: Flags [S.], cksum 0x30aa (correct), seq 3655318959, ack 2269409708, win 65160, options [mss 1460,sackOK,TS val 275303459 ecr 2826982365,nop,wscale 7], length 0
^C
14 packets captured
22 packets received by filter
0 packets dropped by kernel

So, that's 9 returned packets for 9 firewall blocks. By the way in my firewall it gets blocked by the default rule:

Code Select
site1-opnsense # grep 02f4bab031b57d1e30553ce08e0ec131 /tmp/rules.debug
block in log inet from {any} to {any} label "02f4bab031b57d1e30553ce08e0ec131" # Default deny / state violation rule
block in log inet6 from {any} to {any} label "02f4bab031b57d1e30553ce08e0ec131" # Default deny / state violation rule

The question is, why isn't there a "igb3,match,pass,out,.....,10.1.8.27,10.2.4.43,36596,22,....." on the second line of the firewall log? I guess the lack of a match is why the return 'in' traffic is not tracked as established and getting blocked?

All my rules are floating and should apply to the "IPSec" interface, and indeed I see them in the UI in the "Floating rules" drilldown list.

What am I misunderstanding about how PF works with enc0?
Today at 02:41:18 PM #1
To answer my own question:

It looks to me like a problem with connection tracking.

The ICMP traffic works because it just has a rule to allow in any direction, no state needed.

The SSH outbound fails because it disappears into the IPsec SPD _before_ it hits enc0, and so gets tracked from lagg0_vlan1018, so the returning traffic that _does_ appear on enc0 has no matching state and gets dropped unless I put some funky firewall rules in place.

Of course the inbound SSH gets tracked on the enc0 interface and so is allowed.

At least this is my theory. That is the theory that I have and which is mine and what it is, too.

The solution in the end was to use VTI which _does_ enter and exit a virtual interface and so the firewall rules are all good.
Today at 03:08:15 PM #2
What is Firewall: Settings: Advanced: Bind states to interface set to? Read the help text. By default states should be global/floating.
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)
Today at 03:36:21 PM #3
Yep, it's unchecked. I thought this should've helped me too.
Print
Go Up Pages1
User actions