[SOLVED] Problem with firewall - packets don't mach rules?!?!

[borys.ohnsorge]

Hi,

Setup:
HA cluster master/backup (pfsync)

Version:
OPNsense 24.7.9_1-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15

Problem:
Packet don't mach firewall rules on WAN interface and NAT (Outbound).


Packet flow:
Host A on AWS 10.50.10.10 <-> IPSec Tunnel <-> (10.9.255.1) OPNSense (WAN x.x.x.x ip that has full routing to 172.30.1.0/24) <-> int-fw (has route to 10.50.10.10 trough OPNsense advertised by BGP) <-> Host B 172.30.1.201 (int-fw as a def gw)

The packet is sent from Host A to address 10.9.255.1 and destination port 8101. On OPNSense the port forwarding rule is set to 172.30.1.201 and destination port 8101. After reaching 172.30.1.201 the reply is sent back. It reaches the WAN interface of OPNsesne where it should be passed through the rule on the WAN and then NATed (Outbound) to 10.9.255.1 and tunneled to Host A.

On OPNSense enc0 interface I can see packet coming in:

Code Select Expand

root@vpn-opn-1:~ # tcpdump -nienc0 host 10.50.10.10
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enc0, link-type ENC (OpenBSD encapsulated IP), snapshot length 262144 bytes
12:25:32.844266 (authentic,confidential): SPI 0xc231abbe: IP 10.50.10.10.41646 > 10.9.255.1.8101: Flags [S], seq 2403085221, win 62727, options [mss 1375,sackOK,TS val 2428826062 ecr 0,nop,wscale 7], length 0
12:25:43.840594 (authentic,confidential): SPI 0xc231abbe: IP 10.50.10.10.44598 > 10.9.255.1.8101: Flags [S], seq 2402410318, win 62727, options [mss 1375,sackOK,TS val 2428837058 ecr 0,nop,wscale 7], length 0
12:25:44.844619 (authentic,confidential): SPI 0xc231abbe: IP 10.50.10.10.44598 > 10.9.255.1.8101: Flags [S], seq 2402410318, win 62727, options [mss 1375,sackOK,TS val 2428838062 ecr 0,nop,wscale 7], length 0
^C
3 packets captured
1289 packets received by filter
0 packets dropped by kernel


On Host B I can see packet from Host A, and response is send:

Code Select Expand

root@ingester:~# tcpdump -ni bond0.1400 port 8101
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0.1400, link-type EN10MB (Ethernet), capture size 262144 bytes
12:30:08.390725 IP 10.50.10.10.52000 > 172.30.1.201.8101: Flags [S], seq 1619491635, win 62727, options [mss 1375,sackOK,TS val 2429101425 ecr 0,nop,wscale 7], length 0
12:30:08.390796 IP 172.30.1.201.8101 > 10.50.10.10.52000: Flags [S.], seq 2658258262, ack 1619491636, win 65160, options [mss 1460,sackOK,TS val 3474148164 ecr 2429101425,nop,wscale 7], length 0
12:30:09.428097 IP 10.50.10.10.52000 > 172.30.1.201.8101: Flags [S], seq 1619491635, win 62727, options [mss 1375,sackOK,TS val 2429102462 ecr 0,nop,wscale 7], length 0
12:30:09.428144 IP 172.30.1.201.8101 > 10.50.10.10.52000: Flags [S.], seq 2674466878, ack 1619491636, win 65160, options [mss 1460,sackOK,TS val 3474149201 ecr 2429102462,nop,wscale 7], length 0
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
root@ingester:~#

I can see response on WAN interface of OPNsense:

Code Select Expand

root@vpn-opn-1:~ # tcpdump -nivtnet0 host 10.50.10.10
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:31:56.379511 IP 10.50.10.10.38038 > 172.30.1.201.8101: Flags [S], seq 2684226038, win 62727, options [mss 1375,sackOK,TS val 2429209596 ecr 0,nop,wscale 7], length 0
12:31:56.380105 IP 172.30.1.201.8101 > 10.50.10.10.38038: Flags [S.], seq 2006560483, ack 2684226039, win 65160, options [mss 1460,sackOK,TS val 3474256336 ecr 2429209596,nop,wscale 7], length 0
12:31:57.404419 IP 10.50.10.10.38038 > 172.30.1.201.8101: Flags [S], seq 2684226038, win 62727, options [mss 1375,sackOK,TS val 2429210622 ecr 0,nop,wscale 7], length 0
12:31:57.404941 IP 172.30.1.201.8101 > 10.50.10.10.38038: Flags [S.], seq 2022573573, ack 2684226039, win 65160, options [mss 1460,sackOK,TS val 3474257361 ecr 2429210622,nop,wscale 7], length 0
12:31:58.430947 IP 172.30.1.201.8101 > 10.50.10.10.38038: Flags [S.], seq 2022573573, ack 2684226039, win 65160, options [mss 1460,sackOK,TS val 3474258387 ecr 2429210622,nop,wscale 7], length 0
12:32:00.446930 IP 172.30.1.201.8101 > 10.50.10.10.38038: Flags [S.], seq 2022573573, ack 2684226039, win 65160, options [mss 1460,sackOK,TS val 3474260403 ecr 2429210622,nop,wscale 7], length 0
^C
6 packets captured
4609 packets received by filter
0 packets dropped by kernel
root@vpn-opn-1:~ #

Now it should passed by WAN rule and NATed (Outbound) back to 10.9.255.1 and go through tunnel, but this does not happen.



What am I doing wrong because I'm out of ideas??

[borys.ohnsorge]

Problem was on WAN default outgoing rule. After adding individual outgoing rule with reply-to everything start working.