(solved) Port forward on tier 2 wan interface not working

[palillo]

Hi,

(solved) In order to make port forward work on other interfaces different when there is more than 1 WAN interfce, you have to go to Firewall->Settings->Advanced and make sure to tick enable "Disable force gateway" option.  If not, port forward may reply using the wrong gateway and connections will never get established!

It took me lots of hours to figure this out so I hope this may be useful to others!

------------------------------------------------------------------------------------------------

I am currently using OPNSENSE 21.7.2 (lastest as of this writing)

I have 2 WAN connections from 2 different ISPs I am using in a failover mode configuration via a GATEWAY GROUP and Policy routes. I have set up this following carefully the multi-wan tutorial and lots of reading (no trouble on this)

WAN1 (tier 1) users a static private IP (192.168.101.2/24) and is NATed from the ISP fiber modem but, incoming connections are not allow from the ISP so it can't be used to port forward traffic on this interface.

WAN2 (tier 2) is. VDSL with private IP (10.0.2.254/24) but ISP has set a 1:1 NAT from a public IP address so port forward works perfectly on this interface.

LAN firewall rules has its last rule for policy routing to the gateway group.

- Sticky connection is enable
- Port reflection is enable

Now the problem, when I port forward, lets say TCP port 22000 on WAN2 address to an internal IP on my lan on port 22 (ssh service) I see incoming packet arrive perfectly to the internal server and then it responds correctly by sending outgoing packet back from port 22 to originating external machine port.

I checked this with tcpdump on internal ssh server.

The thing is, connection is not being established.

I thought, well, maybe the reply packet (outgoing) is going out on the TIER 1 (WAN1) interface and since the public address is different, it will simply be dropped.

Then, I created a policy routing just for the internal ssh server to force it to go out on the WAN2 (Tier2) interface just in case.  This works just fine as when I ssh into an external server, I can see connection is comingo from WAN2 public IP address.

Then, I tried to connect again from external host to port 2200 on WAN2 public address, packet arrives on internal ssh server and it is responded accordingly, but.... connection never get established!

If I traceroute from ssh internal server to the outside world, I can see traffic goes out on WAN2 as expected.....

I have tried rules on the WAN2/LAN/UNBOUND to allow outgoing traffic to the incomingo IP address, or allowing all out traffic from the internal ssh server (which is redundant), but I have not been able to make port forward to work.

Any thoughts or recommendations on this ? I am very frustrated as I have been trying for days to set this up.  I have been working with Sophos UTM/XG for years, and this is fairly easy to accomplish with them.

Following, you can see on the internal ssh server (tankmonr.local) how packets are received and answered from public-ip.com (changed from original name).  I notice that on response packet there is an (incorrect) message but can't find what it means.  It seems that this incorrect message does not means anything harmful as it is shown on normal ssh connections as well.

Any advise or recommendation is greatly appreciated.

Code: [Select]

root@tankmonr:~# cat borrar
tcpdump  -i eth0 -vv host public-ip.com

root@tankmonr:~# tcpdump  -i eth0 -vv host gammadvpn.telemogroup.com
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:28:20.793196 IP (tos 0x10, ttl 51, id 27165, offset 0, flags [DF], proto TCP (6), length 60)
    public-ip.com.33510 > tankmonr.local.ssh: Flags [S], cksum 0xb03c (correct), seq 3658402422, win 64240, options [mss 1460,sackOK,TS val 360069278 ecr 0,nop,wscale 7], length 0
22:28:20.793396 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    tankmonr.local.ssh > public-ip.com.33510: Flags [S.], cksum 0x276b (incorrect -> 0x4d0a), seq 3969669530, ack 3658402423, win 65160, options [mss 1460,sackOK,TS val 1950328082 ecr 360069278,nop,wscale 7], length 0
22:28:21.806899 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    tankmonr.local.ssh > public-ip.com.33510: Flags [S.], cksum 0x276b (incorrect -> 0x4914), seq 3969669530, ack 3658402423, win 65160, options [mss 1460,sackOK,TS val 1950329096 ecr 360069278,nop,wscale 7], length 0
22:28:21.808305 IP (tos 0x10, ttl 51, id 27166, offset 0, flags [DF], proto TCP (6), length 60)
    public-ip.com.33510 > tankmonr.local.ssh: Flags [S], cksum 0xac45 (correct), seq 3658402422, win 64240, options [mss 1460,sackOK,TS val 360070293 ecr 0,nop,wscale 7], length 0
22:28:21.808383 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    tankmonr.local.ssh > public-ip.com.33510: Flags [S.], cksum 0x276b (incorrect -> 0x4913), seq 3969669530, ack 3658402423, win 65160, options [mss 1460,sackOK,TS val 1950329097 ecr 360069278,nop,wscale 7], length 0
22:28:23.822937 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    tankmonr.local.ssh > public-ip.com.33510: Flags [S.], cksum 0x276b (incorrect -> 0x4134), seq 3969669530, ack 3658402423, win 65160, options [mss 1460,sackOK,TS val 1950331112 ecr 360069278,nop,wscale 7], length 0
22:28:27.854903 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    tankmonr.local.ssh > public-ip.com.33510: Flags [S.], cksum 0x276b (incorrect -> 0x3174), seq 3969669530, ack 3658402423, win 65160, options [mss 1460,sackOK,TS val 1950335144 ecr 360069278,nop,wscale 7], length 0
22:28:36.046917 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)


Thanks in advance.

[fgendorf]

Hi, it seams similar to my https://forum.opnsense.org/index.php?topic=24766.0 , I have two wan and wireguard vpn just accept connections on active gateway, that is tier 1