[SOLVED] After migration port forwarding does not work anymore

[alh]

We moved from ESXi to Proxmox by restoring our config. We now experience issues with port forwarding. If we do a packet capture we can see

- the request hitting the WAN interface
- the request leaving the LAN interface to the target
- the request hitting the target
- the reply leaving the target for the sender

However, the reply never reaches the sender. There is nothing in the firewall logs whatsoever. We also deleted the port forwarding and tried to set it up again but to no avail. The gateways are correctly set on all devices and all other services work just fine.

Any help appreciated.

[mimugmail]

You see the reply on LAN incoming to the Firewall?

[alh]

No, that is the weird part. However, the target host has the LAN of the firewall as default gateway.

UPDATE: the reply is hitting the firewall, see below

[alh]

And the port forward doesn't work for any target (even the firewall itself, e. g. we had a setup with OpenVPN running on localhost).

[alh]

I searched some more and it seems that port randomization is messing up the reply! Why does it mess with an established connection???

- this is the packet capture in opensense on LAN

Code Select Expand


No. Time Source Destination Protocol Length Info
16 2.417953 <clientip> 10.0.1.100 TCP 78 57719 → 8443 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1452 WS=64 TSval=219446651 TSecr=0 SACK_PERM=1
17 2.418656 10.0.1.100 <clientip> TCP 74 8443 → 57719 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=3573347765 TSecr=219446651 WS=128
22 3.419790 <clientip> 10.0.1.100 TCP 78 [TCP Retransmission] 57719 → 8443 [SYN] Seq=0 Win=65535 Len=0 MSS=1452 WS=64 TSval=219447651 TSecr=0 SACK_PERM=1
23 3.420410 10.0.1.100 <clientip> TCP 74 [TCP Retransmission] 8443 → 57719 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=3573348767 TSecr=219446651 WS=128
38 4.425322 <clientip> 10.0.1.100 TCP 78 [TCP Retransmission] 57719 → 8443 [SYN] Seq=0 Win=65535 Len=0 MSS=1452 WS=64 TSval=219448651 TSecr=0 SACK_PERM=1
39 4.425996 10.0.1.100 <clientip> TCP 74 [TCP Retransmission] 8443 → 57719 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=3573349772 TSecr=219446651 WS=128


- this is the packet capture in opensense on WAN

Code Select Expand


No. Time Source Destination Protocol Length Info
14 2.417808 <clientip> <publicip> TCP 78 57719 → 8443 [SYN, ECN, CWR] Seq=0 Win=65535 Len=0 MSS=1452 WS=64 TSval=219446651 TSecr=0 SACK_PERM=1
15 2.418959 <publicip> <clientip> TCP 74 38238 → 57719 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=3573347765 TSecr=219446651 WS=128
16 2.434824 <clientip> <publicip> TCP 60 57719 → 38238 [RST, ACK] Seq=1041084811 Ack=1 Win=0 Len=0
35 3.419890 <clientip> <publicip> TCP 78 [TCP Retransmission] 57719 → 8443 [SYN] Seq=0 Win=65535 Len=0 MSS=1452 WS=64 TSval=219447651 TSecr=0 SACK_PERM=1
36 3.420647 <publicip> <clientip> TCP 74 [TCP Retransmission] 38238 → 57719 [SYN, ACK, ECN] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=3573348767 TSecr=219446651 WS=128


[alh]

Managed to fix this with additional rules in Firewall > NAT > Outbound configuring static ports. However, I still fail to understand this behaviour and if my memory serves me correctly this was not necessary in 20.1 and earlier.

Example rule:

Code Select Expand


Interface Source Source Port Destination Destination Port NAT Address NAT Port Static Port
WAN any tcp/8443 * tcp/ * Interface address * YES


[alh]

Comparing this to another OPNsense (20.1.7 compared to 20.1.8) I get the same behaviour.

However, it seems that this only occurs if there is no destination port translation. If there is, then the rules are not needed. Is this intended behaviour or a bug?

Example:

- DNAT tcp/80 to tcp/80: additional outbound nat rule with static port needed
- DNAT tcp/80 to tcp/8080: NO additional outbound nat rule with static port needed

[alh]

I believe this to be a bug which I reported here https://github.com/opnsense/core/issues/5087

[mimugmail]

Never .. ever .. use any as a source in outbound nat. It will rewrite reply packets of your port forward. Change source to your lan

[alh]

Thanks a lot! That was indeed the issue. Still surprised that it was working before, didn't touch those rules in a while...