port forwarding

[jody]

I'm running opnsense 25.7.1 on a dec750
i have fiber from my ISP come into the house, connected to the dec750, I have ipv4 (and ipv6 but not using it on local lan) external IP, my internal lan is on 192.168.2.0/24, internet works fine, i also have port forwarding working e.g. for lan client *.3, also for the *.10 one, it does work.

Here my NAT port forward config:



and here my Rules WAN:


Now I want to add another port forward rule e.g. forward incoming port 36570 to internal ip *.15 and port 8080 (e.g. running a webserver), I used the "clone" function next to the existing port 443 rule



and



I applied the changes, even restarted the router but the forward rule does not work. What am I missing here?

[meyergru]

Probably your OpnSense web UI is running on the same port. You can change it under "System: Settings: Administration". As an alternative, you can also remove the WAN interface from the listening interfaces.

[jody]

no, that is not the problem, because when I ask it to forward port 36570 to 192.168.2.3 or 192.168.2.10 it does work, the port is forwarded, so the problem must be something different

[viragomann]

To verify if the packet on port 36570 even arrive, start a packet capture on the WAN, filtering for port 36570, then try to access it from outside.

[meyergru]

If the port is being forwarded to another machine, then it must be something on the new target:

1. It does not listen on the target port.
2. It does not allow access outside of its own network.
3. It is missing the correct gateway to route packets back.

[jody]

thank you for the suggestion, I have some tcpdump commands on my firewall and then I tried to establish a connection from outside,

Code Select Expand

tcpdump -i pppoe0 port 36570
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), snapshot length 262144 bytes
13:55:49.054018 IP 95-44-98-92.ftth.glasoperator.nl.52494 > 181.22.134.122.36570: Flags [S], seq 1266372702, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 2228375155 ecr 0,sackOK,eol], length 0
13:55:50.053916 IP 95-44-98-92.ftth.glasoperator.nl.52494 > 181.22.134.122.36570: Flags [S], seq 1266372702, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 2228376156 ecr 0,sackOK,eol], length 0
13:55:51.055410 IP 95-44-98-92.ftth.glasoperator.nl.52494 > 181.22.134.122.36570: Flags [S], seq 1266372702, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 2228377158 ecr 0,sackOK,eol], length 0
13:55:52.057047 IP 95-44-98-92.ftth.glasoperator.nl.52494 > 181.22.134.122.36570: Flags [S], seq 1266372702, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 2228378159 ecr 0,sackOK,eol], length 0
13:55:53.059336 IP 95-44-98-92.ftth.glasoperator.nl.52494 > 181.22.134.122.36570: Flags [S], seq 1266372702, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 2228379160 ecr 0,sackOK,eol], length 0
13:55:54.060072 IP 95-44-98-92.ftth.glasoperator.nl.52494 > 181.22.134.122.36570: Flags [S], seq 1266372702, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 2228380161 ecr 0,sackOK,eol], length 0
13:55:56.060877 IP 95-44-98-92.ftth.glasoperator.nl.52494 > 181.22.134.122.36570: Flags [S], seq 1266372702, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 2228382162 ecr 0,sackOK,eol], length 0
13:56:00.065529 IP 95-44-98-92.ftth.glasoperator.nl.52494 > 181.22.134.122.36570: Flags [S], seq 1266372702, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 2228386163 ecr 0,sackOK,eol], length 0
^C
8 packets captured
6988 packets received by filter
0 packets dropped by kernel
i thought let's change the ip to *.3 and see what happens when it does work and establishes a connection:

Code Select Expand

tcpdump -i pppoe0 port 36570
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), snapshot length 262144 bytes
13:59:04.706899 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [S], seq 3564263277, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 3991835895 ecr 0,sackOK,eol], length 0
13:59:04.707548 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [S.], seq 891614574, ack 3564263278, win 65160, options [mss 1460,sackOK,TS val 1648323393 ecr 3991835895,nop,wscale 7], length 0
13:59:04.762413 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [.], ack 1, win 2052, options [nop,nop,TS val 3991835951 ecr 1648323393], length 0
13:59:04.762909 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [P.], seq 1:22, ack 1, win 2052, options [nop,nop,TS val 3991835951 ecr 1648323393], length 21
13:59:04.763362 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [.], ack 22, win 509, options [nop,nop,TS val 1648323449 ecr 3991835951], length 0
13:59:04.834003 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [P.], seq 1:41, ack 22, win 509, options [nop,nop,TS val 1648323520 ecr 3991835951], length 40
13:59:04.888644 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [.], ack 41, win 2052, options [nop,nop,TS val 3991836077 ecr 1648323520], length 0
13:59:04.889089 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [P.], seq 41:1177, ack 22, win 509, options [nop,nop,TS val 1648323575 ecr 3991836077], length 1136
13:59:04.891651 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [.], seq 22:1390, ack 41, win 2052, options [nop,nop,TS val 3991836080 ecr 1648323520], length 1368
13:59:04.891657 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [P.], seq 1390:1590, ack 41, win 2052, options [nop,nop,TS val 3991836080 ecr 1648323520], length 200
13:59:04.892093 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [.], ack 1390, win 499, options [nop,nop,TS val 1648323578 ecr 3991836080], length 0
13:59:04.892098 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [.], ack 1590, win 498, options [nop,nop,TS val 1648323578 ecr 3991836080], length 0
13:59:04.941646 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [.], ack 1177, win 2035, options [nop,nop,TS val 3991836131 ecr 1648323575], length 0
13:59:04.961532 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [P.], seq 1590:2798, ack 1177, win 2048, options [nop,nop,TS val 3991836151 ecr 1648323578], length 1208
13:59:04.961946 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [.], ack 2798, win 498, options [nop,nop,TS val 1648323648 ecr 3991836151], length 0
13:59:04.983387 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [.], seq 1177:2545, ack 2798, win 508, options [nop,nop,TS val 1648323669 ecr 3991836151], length 1368
13:59:04.983392 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [P.], seq 2545:2741, ack 2798, win 508, options [nop,nop,TS val 1648323669 ecr 3991836151], length 196
13:59:05.037549 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [.], ack 2545, win 2027, options [nop,nop,TS val 3991836227 ecr 1648323669], length 0
13:59:05.037561 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [.], ack 2741, win 2024, options [nop,nop,TS val 3991836227 ecr 1648323669], length 0
13:59:10.052319 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [P.], seq 2798:2814, ack 2741, win 2048, options [nop,nop,TS val 3991841237 ecr 1648323669], length 16
13:59:10.094741 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [.], ack 2814, win 508, options [nop,nop,TS val 1648328781 ecr 3991841237], length 0
13:59:10.152864 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [P.], seq 2814:2858, ack 2741, win 2048, options [nop,nop,TS val 3991841342 ecr 1648328781], length 44
13:59:10.153109 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [.], ack 2858, win 508, options [nop,nop,TS val 1648328839 ecr 3991841342], length 0
13:59:10.153298 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [P.], seq 2741:2785, ack 2858, win 508, options [nop,nop,TS val 1648328839 ecr 3991841342], length 44
13:59:10.209736 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [.], ack 2785, win 2048, options [nop,nop,TS val 3991841399 ecr 1648328839], length 0
13:59:10.210202 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [P.], seq 2858:2926, ack 2785, win 2048, options [nop,nop,TS val 3991841399 ecr 1648328839], length 68
13:59:10.225126 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [P.], seq 2785:2837, ack 2926, win 508, options [nop,nop,TS val 1648328911 ecr 3991841399], length 52
13:59:10.276209 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [.], ack 2837, win 2048, options [nop,nop,TS val 3991841465 ecr 1648328911], length 0
13:59:12.475250 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [P.], seq 2926:3074, ack 2837, win 2048, options [nop,nop,TS val 3991843664 ecr 1648328911], length 148
13:59:12.518682 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [.], ack 3074, win 508, options [nop,nop,TS val 1648331205 ecr 3991843664], length 0
13:59:14.312354 IP 181.22.134.122.36570 > 95-44-98-92.ftth.glasoperator.nl.52644: Flags [P.], seq 2837:2889, ack 3074, win 508, options [nop,nop,TS val 1648332998 ecr 3991843664], length 52
13:59:14.391277 IP 95-44-98-92.ftth.glasoperator.nl.52644 > 181.22.134.122.36570: Flags [.], ack 2889, win 2048, options [nop,nop,TS val 3991845580 ecr 1648332998], length 0
^C
32 packets captured
4488 packets received by filter
0 packets dropped by kernel
when i try tcpdump on my other interfaces ax0 and/or ax1 i get nothing on both accounts

I have have internal webservers running on port 8080 on lan ip *.15 *.142 *.16 *.17 all accessible internally I tried forwarding to those as well but no luck

[viragomann]

So obviously the webserver doesn't respond, if the access is coming from the internet (or outside of your LAN). You can also sniff the traffic on the internal interface to get sure.

meyergru already mentioned the most proper possible reasons for this issue. Did you verify them already?

[jody]

yes, had checked those but there seems to be a gateway problem maybe, running tcpdump on webserver hosts shows the below:

Code Select Expand

tcpdump -i enp116s0 port 8080
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp116s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:31:32.489213 IP _gateway.8080 > aqua.50256: Flags [P.], seq 4290346268:4290346328, ack 1999504314, win 514, options [nop,nop,TS val 415337249 ecr 1851539844], length 60
18:31:32.489259 IP aqua.50256 > _gateway.8080: Flags [P.], seq 1:37, ack 60, win 12703, options [nop,nop,TS val 1851569922 ecr 415337249], length 36
18:31:32.489400 IP _gateway.8080 > aqua.50256: Flags [.], ack 37, win 514, options [nop,nop,TS val 415337249 ecr 1851569922], length 0
aqua is my wireguard interface on host I'm forwarding the port to, all clients (except the one ending on *.3 and *.10) have a vpn working connecting to a vpn server on the internet, I guess that is somehow messing with the gateway?

Code Select Expand

ip route
default via 192.168.2.1 dev enp116s0 proto static
10.110.19.0/24 dev aqua.wg proto kernel scope link src 10.110.19.20
192.168.2.0/24 dev enp116s0 proto kernel scope link src 192.168.2.23
[root@aqua funky1]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.2.1     0.0.0.0         UG    0      0        0 enp116s0
10.110.19.0     0.0.0.0         255.255.255.0   U     0      0        0 aqua.wg
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 enp116s0


[viragomann]

aqua is my wireguard interface

So you're running a Wireguard VPN on the webserver itself?
And the tcpdump should tell me, that responses go to the WG IP?

I guess, that what you see here, belongs to a different connection, but hard to tell due to the IP obscuring.

[jody]

wireguard "server" is on the internet, pretty much all hosts on 192.168.2.0/24 are a wireguard "client" (except *.3 and *.10), so 192.168.2.15 is also a wireguard client and also hosts a webserver and other services
to test if wireguard is causing the issue i disabled the wireguard client on *.15 and I do get a connection, so indeed wireguard screws up the gateway setting?
how would i get that fixed?

[meyergru]

So, you essentially say that *.3 and *.10 are not wireguard clients, but *.15 is and the latter does not work with port forwarding.

I assume all wireguard clients (which seems to be most of the subnet) have a gateway over wireguard, maybe unbeknownst to you.

You should probably segment your network in a way where you separate out servers that should be available via port-forwarding and handle that "DMZ" subnet differently, instead of setting rules for specific IPs within your LAN subnet - which, apparently, goes wrong if you forget one IP.

[jody]

thx meyergru - I found out that when I set in the wireguard client under the [interface] section "Table = off" which I guess prevents wireguard from modifying the routing table, the port forwarding does work again - jeez networking is hard - i will look into your advise and see if/how i can segment my networks