Unable to get port forwarding working!

[Cynopolis]

Hello everyone,
I'm very new to OPNsense and just started using it last week. I've got the basic setup working and can access the internet through my OPNsense router. I previously had several services that I forwarded through my old router and I want to continue forwarding them in OPNsense.

Most of the services go through NGINX which has the IP address: 192.168.0.76 and uses port 9110 for HTTP and 9112 for HTTPS.
I configured OPNsense NAT port forwarding as follows:
----------------
Aliases:
Name: truenas
Type: Host(s)
Content: 192.168.0.76
------------
Name: NGINX_HTTP_Port
Type: Port(s)
Content 9110
------------
Name: NGINX_HTTPS_Port
Type: Port(s)
Content 9112
-------------------
NAT > Port Forward:
Interface: WAN
TCP/IP Version: IPv4
Destination: WAN Address
Destination Port Range: HTTP to HTTP
Redirect target IP: truenas
Redirect target port: NGINX_HTTP_Port
NAT reflection: Enable
Filter rule association: Create associated rule

The above configuration was copied and modified accordingly for the https port.
----------------
Rules > WAN
Protocol: IPv4 TCP
Source: *
Port *
Destination: truenas
Port: NGINX_HTTP_Port
Gateway: *
Schedule *

There is another entry here for HTTPS as well. Both of these were automatically added by OPNsense when I configured the port forwarding settings.
--------------------
Settings > Advanced:
Reflection for port forwards: enabled
Reflection for 1:1: disabled
Automatic outbound NAT for Reflection: enabled
--------------------

When I try to access one of my sites such as test.example.com while connected to the WiFi, I have no issue accessing it. However, when I try to access the site from my phone using data I just get ERR_CONNECTION_TIMED_OUT.

I've been pulling my hair out over this for days trying to get this working and I've perused many other posts on here which don't seem to have quite the same issues. Any help would really be appreciated, thanks!

[meyergru]

How is your OpnSense attached to the internet? Do you put it behind your router? That is a router-behind-router scenario which is much harder to set up correctly. The preferred way is to use a modem or ONT in bridge mode, if at all possible.

Assuming your previous router can do port forwarding, we can rule out CG-NAT, right?

[cookiemonster]

Code Select Expand

port 9110 for HTTP and 9112 for HTTPS.

Code Select Expand

NAT > Port Forward:
Interface: WAN
TCP/IP Version: IPv4
Destination: WAN Address
Destination Port Range: HTTP to HTTP
Redirect target IP: truenas
Redirect target port: NGINX_HTTPS_Port
NAT reflection: Enable
Filter rule association: Create associated rule
Also, is it possible you are forwarding port 80 to 9112 instead of 9110 or just a typo?

[Cynopolis]

Also, is it possible you are forwarding port 80 to 9112 instead of 9110 or just a typo?

Apologies, that's just a typo. It was pretty late when I wrote this so I accidentally added an extra "s".

How is you OpnSense attached to the internet? Do you put it behind your router?

OPNSense is acting as my router and its wan port is connected directly to my modem via an Ethernet cable. The LAN port is then connected to a switch that goes to every other device. My former router now just acts as a dumb wifi AP and is connected to the switch like all of the other devices.

Assuming your previous router can do port forwarding, we can rule out CG-NAT, right?

Yeah I've ruled out CG-NAT because I was successfully port forwarding until I switched to OPNSense. The only thing I've changed in my setup is the router so I'm pretty confident that's where the problem lies.

[meyergru]

The NAT rules look good, lathough I prefer to use "this firewall" as destination address. You used "Create associated rule" instead of just "pass" - this bears the risk that the rules are evaluated after other rules on the same interface or floating rules, so they might not get used.

Apart from that, you can check if the packets are passed by using tcpdump on your LAN interface.

If that works, then there may be other problems like the default gateway / route on your Truenas (maybe your old router had another IP?) or additional limitations like allowed IP ranges in Nginx or firewall rules on Truenas itself.

[Cynopolis]

I will change the rule to "pass" temporarily to see if that fixes anything. I'll also get logs from tcpdump on my LAN interface when I get home. I don't think there's an issue with my TrueNAS or NGINX config since the OPNsense router was a 1:1 replacement with my old router and has taken the same IP address. (192.168.0.1)

[Cynopolis]

Apart from that, you can check if the packets are passed by using tcpdump on your LAN interface.

For tcpdump I simultaneously logged LAN and WAN packets and looked for the same packet getting passed from WAN to LAN. I saw some SYN packets that appeared to be successfully passed from WAN:443 to truenas:9112. I wish I could upload the wireshark logs, but I'm worried about privacy since the logs contain my public IP address. I'm pretty new to analyzing these kinds of logs.

I also started looking at Firewall > Log Files > Live View to see if I could find any packets that were blocked trying to access port 80 or 443 on my external IP address. In my port forwarding rules, I enabled logging packets handled by the port forward rules and then in the firewwall live view I could see tons of packets getting redirected as they were coming to my WAN IP address on port 80/443. I didn't see any packets getting blocked if they accessed my WAN IP on port 80 or 443

[EricPerl]

You can start with sharing the packet captures on the LAN side.
You'd at least want to see traffic coming back...

[JamesFrisch]

Yeah I've ruled out CG-NAT because I was successfully port forwarding until I switched to OPNSense. The only thing I've changed in my setup is the router so I'm pretty confident that's where the problem lies.

Sorry, but unless you spoofed the MAC address of your old router, this isn't good enough.

I know ISPs that give you CG-NAT and only hand out a real IPv4 when the user configures Port Forwarding in their customer center (router is not locally configurable, only over the ISP webpage).


Here is how you can test it in under 5min:
https://github.com/jameskimmel/opinions_about_tech_stuff/blob/main/network%20stuff/CG-NAT.md

OPNSense is acting as my router and its wan port is connected directly to my modem via an Ethernet cable.

So your modem is in bridge mode, right?
Test from above will show this.



Maybe to make this a little bit easier to troubleshoot, could you just create a new rule for port http (80) and see if certbot (I assume you use certbot?) is able to get a cert?

[Cynopolis]

This isn't a particularly satisfying resolution, but it did end up fixing the problem. I factory reset my OPNsense router, set it up from scratch, and re-created the port forwarding rules. Now everything works as expected! There must have been some rule I initially misconfigured which was causing an issue.

Thank you to everyone who gave advice on this I appreciate the help you provided!