Hi all,
I have already searched the forum, but unfortunately I haven't found anything that could solve my problem.
My setup:
fritzbox: internal 192.168.9.254, exposed host to 192.168.9.1
opnsense CARP WAN IP 192.168.9.1
opnsense CARP DMZ IP 10.66.6.1
ftp server 10.66.6.3
port forward rule:
interface: WAN
destination: 192.168.9.1 (CARP WAN IP)
destination port range: FTP
redirected target: 10.66.6.3
redirected target port: FTP
Unfortunately, no package is arriving at the FTP server.
When I connect the FTP server to the network between Fritz and OPNsense (192.168.9.3) for testing purposes and set the exposed host on Fritz to 192.168.9.3, everything works, meaning that forwarding on Fritz is functioning.
I also unchecked the two boxes for 'Block private networks' and 'Block bogon networks' on the WAN interface, as we have RFC1918 addresses on both sides of the opnsense cluster.
the NAT rule created a rule for the WAN interface::
source: *
port: 21 (FTP)
destination: 10.66.6.3
gateway: *
Version: OPNsense 25.7.6-amd64
Ping between OPNsense and Fritzbox:
root@OPNsense1:~ # ping 192.168.99.254
PING 192.168.99.254 (192.168.99.254): 56 data bytes
64 bytes from 192.168.99.254: icmp_seq=0 ttl=64 time=0.069 ms
64 bytes from 192.168.99.254: icmp_seq=1 ttl=64 time=0.070 ms
64 bytes from 192.168.99.254: icmp_seq=2 ttl=64 time=0.068 ms
^C
--- 192.168.99.254 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.068/0.069/0.070/0.001 ms
Ping between OPNsense and FTP-Server:
root@OPNsense1:~ # ping 10.66.6.3
PING 10.66.6.3 (10.66.6.3): 56 data bytes
64 bytes from 10.66.6.3: icmp_seq=0 ttl=64 time=0.175 ms
64 bytes from 10.66.6.3: icmp_seq=1 ttl=64 time=0.137 ms
64 bytes from 10.66.6.3: icmp_seq=2 ttl=64 time=0.145 ms
64 bytes from 10.66.6.3: icmp_seq=3 ttl=64 time=0.202 ms
64 bytes from 10.66.6.3: icmp_seq=4 ttl=64 time=0.150 ms
64 bytes from 10.66.6.3: icmp_seq=5 ttl=64 time=0.138 ms
64 bytes from 10.66.6.3: icmp_seq=6 ttl=64 time=0.134 ms
^C
--- 10.66.6.3 ping statistics ---
7 packets transmitted, 7 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.134/0.154/0.202/0.023 ms
tcpdump on WAN if, from internet server telnet to port 21 of Fritzbox externally
root@OPNsense1:~ # tcpdump -i igb0 -n src host 138.201.203.161 and port 21
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:25:43.352331 IP 138.201.203.161.38174 > 192.168.9.1.21: Flags [S], seq 1231877551, win 64240, options [mss 1452,sackOK,TS val 1634746021 ecr 0,nop,wscale 7], length 0
13:25:44.406669 IP 138.201.203.161.38174 > 192.168.9.1.21: Flags [S], seq 1231877551, win 64240, options [mss 1452,sackOK,TS val 1634747076 ecr 0,nop,wscale 7], length 0
13:25:45.431783 IP 138.201.203.161.38174 > 192.168.9.1.21: Flags [S], seq 1231877551, win 64240, options [mss 1452,sackOK,TS val 1634748101 ecr 0,nop,wscale 7], length 0
13:25:46.454730 IP 138.201.203.161.38174 > 192.168.9.1.21: Flags [S], seq 1231877551, win 64240, options [mss 1452,sackOK,TS val 1634749124 ecr 0,nop,wscale 7], length 0
13:25:47.479088 IP 138.201.203.161.38174 > 192.168.9.1.21: Flags [S], seq 1231877551, win 64240, options [mss 1452,sackOK,TS val 1634750148 ecr 0,nop,wscale 7], length 0
13:25:48.502637 IP 138.201.203.161.38174 > 192.168.9.1.21: Flags [S], seq 1231877551, win 64240, options [mss 1452,sackOK,TS val 1634751172 ecr 0,nop,wscale 7], length 0
13:25:50.550939 IP 138.201.203.161.38174 > 192.168.9.1.21: Flags [S], seq 1231877551, win 64240, options [mss 1452,sackOK,TS val 1634753220 ecr 0,nop,wscale 7], length 0
13:25:54.582609 IP 138.201.203.161.38174 > 192.168.9.1.21: Flags [S], seq 1231877551, win 64240, options [mss 1452,sackOK,TS val 1634757252 ecr 0,nop,wscale 7], length 0
13:25:54.617809 IP 138.201.203.161.21 > 192.168.9.1.15052: Flags [S.], seq 2802514873, ack 3951982751, win 65160, options [mss 1460,sackOK,TS val 1759046142 ecr 8452315,nop,wscale 7], length 0
13:25:54.700266 IP 138.201.203.161.21 > 192.168.9.1.15052: Flags [P.], seq 1:49, ack 1, win 510, options [nop,nop,TS val 1759046226 ecr 8452343], length 48: FTP: 220 ProFTPD Server (ProFTPD) [138.201.203.161]
13:25:54.728277 IP 138.201.203.161.21 > 192.168.9.1.15052: Flags [.], ack 7, win 510, options [nop,nop,TS val 1759046254 ecr 8452426], length 0
13:25:54.728296 IP 138.201.203.161.21 > 192.168.9.1.15052: Flags [P.], seq 49:63, ack 8, win 510, options [nop,nop,TS val 1759046254 ecr 8452426], length 14: FTP: 221 Goodbye.
13:25:54.731597 IP 138.201.203.161.21 > 192.168.9.1.15052: Flags [F.], seq 63, ack 8, win 510, options [nop,nop,TS val 1759046255 ecr 8452426], length 0
13:26:02.838641 IP 138.201.203.161.38174 > 192.168.9.1.21: Flags [S], seq 1231877551, win 64240, options [mss 1452,sackOK,TS val 1634765508 ecr 0,nop,wscale 7], length 0
^C
14 packets captured
20050 packets received by filter
0 packets dropped by kernel
tcpdump on DMZ if, identical test
root@OPNsense1:~ # tcpdump -i igb1 -n src host 138.201.203.161 and port 21
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
0 packets captured
197877 packets received by filter
0 packets dropped by kernel
root@OPNsense1:~ # tcpdump -i vlan0.666 -n src host 138.201.203.161 and port 21
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vlan0.666, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
0 packets captured
104706 packets received by filter
0 packets dropped by kernel
OPNsense ifconfig
root@OPNsense1:~ # ifconfig
igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether 20:7c:14:a1:68:2c
inet 192.168.9.11 netmask 0xffffff00 broadcast 192.168.9.255
inet 192.168.9.1 netmask 0xffffff00 broadcast 192.168.9.255 vhid 9
inet6 fe80::227c:14ff:fea1:682c%igb0 prefixlen 64 scopeid 0x1
carp: MASTER vhid 9 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
description: LAN (lan)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether 20:7c:14:a1:68:2d
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: OPT1 (opt1)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether 20:7c:14:a1:68:2e
inet6 fe80::227c:14ff:fea1:682e%igb2 prefixlen 64 scopeid 0x3
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
description: HALink (opt2)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether 20:7c:14:a1:68:2f
inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
options=0
groups: enc
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pfsync0: flags=1000041<UP,RUNNING,LOWER_UP> metric 0 mtu 9000
options=0
syncdev: igb3 syncpeer: 192.168.0.2 maxupd: 128 defer: off version: 1400
syncok: 1
groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
options=0
groups: pflog
vlan0.21: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
description: LAN_VLAN21 (opt6)
options=4000000<MEXTPG>
ether 20:7c:14:a1:68:2d
inet 192.168.21.251 netmask 0xffffff00 broadcast 192.168.21.255
inet 192.168.21.254 netmask 0xffffff00 broadcast 192.168.21.255 vhid 21
groups: vlan
carp: MASTER vhid 21 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
vlan: 21 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.51: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
description: LAN_VLAN51 (opt3)
options=4000000<MEXTPG>
ether 20:7c:14:a1:68:2d
inet 192.168.51.251 netmask 0xffffff00 broadcast 192.168.51.255
inet 192.168.51.254 netmask 0xffffff00 broadcast 192.168.51.255 vhid 51
groups: vlan
carp: MASTER vhid 51 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
vlan: 51 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.52: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
description: LAN_VLAN52 (opt4)
options=4000000<MEXTPG>
ether 20:7c:14:a1:68:2d
inet 192.168.52.251 netmask 0xffffff00 broadcast 192.168.52.255
inet 192.168.52.254 netmask 0xffffff00 broadcast 192.168.52.255 vhid 52
groups: vlan
carp: MASTER vhid 52 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
vlan: 52 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.53: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
description: LAN_VLAN53 (opt5)
options=4000000<MEXTPG>
ether 20:7c:14:a1:68:2d
inet 192.168.53.251 netmask 0xffffff00 broadcast 192.168.53.255
inet 192.168.53.254 netmask 0xffffff00 broadcast 192.168.53.255 vhid 53
groups: vlan
carp: MASTER vhid 53 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
vlan: 53 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.666: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
description: LAN_VLAN666 (opt7)
options=4000000<MEXTPG>
ether 20:7c:14:a1:68:2d
inet 10.66.6.251 netmask 0xffffff00 broadcast 10.66.6.255
inet 10.66.6.1 netmask 0xffffff00 broadcast 10.66.6.255 vhid 66
groups: vlan
carp: MASTER vhid 66 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
vlan: 666 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
wg0: flags=10080c1<UP,RUNNING,NOARP,MULTICAST,LOWER_UP> metric 0 mtu 1420
options=80000<LINKSTATE>
inet 192.168.101.0 netmask 0xffffff00
groups: wg wireguard
nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
vlan0.99: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
description: LAN_VLAN99 (opt8)
options=4000000<MEXTPG>
ether 20:7c:14:a1:68:2d
inet 192.168.99.251 netmask 0xffffff00 broadcast 192.168.99.255
inet 192.168.99.254 netmask 0xffffff00 broadcast 192.168.99.255 vhid 99
groups: vlan
carp: MASTER vhid 99 advbase 1 advskew 0
peer 224.0.0.18 peer6 ff02::12
vlan: 99 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
OPNsense netstat -rn4
root@OPNsense1:~ # netstat -rn4
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 192.168.9.253 UGS igb0
1.1.1.1 192.168.9.253 UGHS igb0
9.9.9.9 192.168.9.254 UGHS igb0
10.66.6.0/24 link#13 U vlan0.666
10.66.6.1 link#5 UHS lo0
10.66.6.251 link#5 UHS lo0
127.0.0.1 link#5 UH lo0
192.168.0.0/24 link#4 U igb3
192.168.0.1 link#5 UHS lo0
192.168.9.0/24 link#1 U igb0
192.168.9.1 link#5 UHS lo0
192.168.9.11 link#5 UHS lo0
192.168.21.0/24 link#9 U vlan0.21
192.168.21.251 link#5 UHS lo0
192.168.21.254 link#5 UHS lo0
192.168.51.0/24 link#10 U vlan0.51
192.168.51.251 link#5 UHS lo0
192.168.51.254 link#5 UHS lo0
192.168.52.0/24 link#11 U vlan0.52
192.168.52.251 link#5 UHS lo0
192.168.52.254 link#5 UHS lo0
192.168.53.0/24 link#12 U vlan0.53
192.168.53.251 link#5 UHS lo0
192.168.53.254 link#5 UHS lo0
192.168.99.0/24 link#15 U vlan0.99
192.168.99.251 link#5 UHS lo0
192.168.99.254 link#5 UHS lo0
192.168.101.0 link#5 UHS lo0
192.168.101.0/24 link#14 U wg0
Any suggestion where I made a mistake?
Slainte
Harald
Quote from: harald99 on October 27, 2025, 01:38:13 PMport forward rule:
interface: WAN
destination: 192.168.9.1 (CARP WAN IP)
destination port range: FTP
Really? FTP without any encryption?
If you really want to do this, you will need the ftp-proxy plugin.
The FTP is with encryption, but on the standard port 21.
I tried the proxy, without luck.
But I will try again.
Tried again, not working also.
Looks like nothing is going thru the OPNsense, which is right, but not for port forwarding.
Any suggestion why the port forward is not working?
Tried a http port forwarding to another vlan (99), same issue, so it must be something common.
is there any rule to disable port forwarding at all?
Wether FTP works or not, I'd expect to see at least the SYN packets on the internal interface, if the port forwarding rule is working. So I searched for your port forwarding rule again.
Quote from: harald99 on October 27, 2025, 01:38:13 PMthe NAT rule created a rule for the WAN interface::
Code Select Expand
source: *
port: 21 (FTP)
destination: 10.66.6.3
gateway: *
You did specify the source port?
It's not necessarily 21 as your TCP dump above shows.
Quote from: viragomann on October 27, 2025, 08:16:29 PMYou did specify the source port?
no, it wouldn't make sense, as the src port is very unlikely same as dst port
I investigated further, looks like it is not rhetorical port forwarding itself.
The WAN interfaces have the addresses 192.168.9.11 (1st), 192.168.9.12 (2nd) and a virtual IP 192.168.9.1.
If the exposed host on the Fritz is set to the virtual IP the port forwarding is not working.
If I set it to the IP of the active OPNsense (1st, 192.168.9.11) the port forwarding is working as expected.
Looks like it has something to do with the CARP Address.
Any suggestions?
The CARP IP address has a special vrrp MAC address range that some devices might not learn correctly.
Here is some more information we added to the docs recently:
https://docs.opnsense.org/manual/how-tos/carp.html#troubleshooting
Your case is most likely a layer 2 issue.
So, one step further:
Forwarding to http and https works.
Unfortunately, FTP still does not.
The FTP is in VLAN666. When I access the FTP from inside (VLAN99) via routing, everything works.
Logging in from outside via port forwarding works, but listing the FTP directory causes it to hang.
Both connections are via FTP with SSL.
I also tried other FTP servers in VLAN666 and VLAN99 with the same result: everything works from the inside (routing), but from the outside (port forwarding), only the login works, not the listing.
FTP sends the IP address and port to use in the control messages, check if there is an unexpected one. NAT does not affect control protocol level control messages.
It works perfectly via the ftp proxy as a reverse proxy, which is a workaround.
I would still like to know why it doesn't work without the listing.