Ports that do not want to be forwarded

[ivica.glavocic]

On my OPNSense v25.1.11, ports 57776 and 57777 can't be forwarded to internal ports. NAT and related firewall rules are OK, ISP is not blocking them. Netstat did not show anything listening on those ports. Any idea why?

[meyergru]

Try "sockstat -l | fgrep 5777" and see which local process is using them currently. However, that may change next time you start those processes.

The reason for this is that you are trying to use two ports that lie in the Ephemeral port range, which you should not. You can set that range via tuneables net.inet.ip.portrange.hilast (65535) and net.inet.ip.portrange.hifirst (49152).

[ivica.glavocic]

Thanks for the quick answer. For testing purposes, in Tunables I have set up net.inet.ip.portrange.hifirst: 59999, port 57777 is still not accessible from outside.
"sockstat -l | fgrep 5777" shows nothing.

[meyergru]

Yet, I am 99% sure that those ports are currently in use. The change of the tuneable only affects newly created processes, probably only after a reboot, IDK.

Allowing only ~500 ephemeral ports is probably a bad idea. The gist of what I wrote is that you should better use ports < 49152, because standards exist for reasons. Other than that, you are on your own.

[ivica.glavocic]

Looks to me as well that those ports are in use, but how can I confirm?

I tried the following:
net.inet.ip.portrange.hifirst: 40000
net.inet.ip.portrange.hilast: 50992
reboot
That gives almost 11.000 ephemeral ports for system to use.

Port 57777 is stil not accessible.

[meyergru]

As stated, standards are there for reasons, I would not change those ports, but rather my destination ports.

FWIW, I just tried to forward port 57776 to another port on a DMZ machine and it worked just fine, so there is no principal problem.

It could also be Crowdsec, Suricata or Zenarmor or any other rules, including implicit (hidden) pf rules that blocks this or maybe your NAT rules are wrong. Maybe you try to rewrite ports on OpnSense itself. Maybe your destination machine does not accept connections on those ports.

You will have to trace the packets down with tcpdump and if they are present, log the packets in OpnSense firewall rules to see what arrives and what goes out.

[Patrick M. Hausen]

FWIW, I just tried to forward port 57776 to another port on a DMZ machine and it worked just fine, so there is no principal problem.

Even if something was listening on that port locally, an inbound NAT port forward rule on WAN would take precedence. pf is independent of the socket API.

I'd question the initial post's "NAT and related firewall rules are OK". Please show those port forward settings and corresponding firewall rules.

Also for a port forward you will not see anything listening in netstat. Again: pf <> socket - separate things.

[ivica.glavocic]

Same thing with pfsense v2.8.0, clean install - TCP port 57777 cannot be forwarded from WAN to LAN test server port 7777.
NAT and firewall rules are 100% correct, multiple checks.
Go figure.

[Patrick M. Hausen]

Show those NAT and firewall rules and we will see if they are 100% correct. Simple as that.

[ivica.glavocic]

Show those NAT and firewall rules and we will see if they are 100% correct. Simple as that.

Here they are.

[bamf]

I do not see any allow rule for port 57777 on the WAN interface.

[ivica.glavocic]

Rules are OK.

TCP WAN 52222 ---> LAN 22 work.
TCP WAN 57777 ---> LAN 7777 does not work.
Same rules, different ports.

Must be something stupid and trivial, just can't figure out what.

[Patrick M. Hausen]

What is Internet_FW_Address? Is it identical with the interface address? Then why did you create that alias and not just use the predefined "Internet address"? Assuming "Internet" is what you named your WAN interface.

[meyergru]

Unless you verified that packets actually arrive on your WAN IP, you cannot be sure that your ISP (or a router in front your OpnSense) does not filter port 57777. And even then, if your LAN server has a local firewall (like many Ubuntu instances do), maybe port 22 is open and port 7777 is not.

It could also be that the server listening on port 7777 is limited to local IPs and by using NAT, you suddenly have source IPs that do not qualify for access. I have seen many services with default configurations acting like that because they are also not configured for encryption. If that is the case, you could use a reverse proxy, which then shows up as the LAN IP of your OpnSense instead of the calling internet IP.

Too little information given to tell...

[ivica.glavocic]

Mistery has been solved.

For testing purposes, I was connecting from one of our fixed public IP address, filtering incoming port forwards on OPNSense and other firewalls only for that public IP.

I really started to bang my head when from that public IP, port forward did not work on Fortinet and Mikrotik, same port. Totally different architectures, yet all have the same problem with that damn 57777 port. Impossible.

It turned out that very same public IP I was connecting from and using for testing, already had 57777 forwarded from outside. It prevented outgoing connections for that port.

As soon as I opened briefly NAT on all firewalls for the entire Internet, and tested from few different IPs, everything worked.

Thanks to everyone for help, you are excellent community, OPNSense is still in the run for our new firewall cluster.