Can not get NGINX to allow connections..

[fbeye]

Hello

I have NPM running on an internal VM/Docker and it works fine but wanted to set it up to be on the opnsense appliance instead and seem to be having some serious issues.
My WAN is static IP and I have a duckdns.org attached to it and verified it is directed as such.
My OPNSense LAN Interface is 172.16.2.1 which is of course the GUI access address..
My OPNSense Port 80 and 443 have been changed so that those 2 ports are free and available for NGINX.
I followed this guide to the T obviously putting in the correct code relevant to mys setup.
I am not sure what I am missing.. And either I am not educated enough in log monitoring to see what I am missing or what, but I see no obvious issues... During the setup everything was set and applied without errors.. The certificates were processed and approved etc.

https://forum.opnsense.org/index.php?topic=24778.msg118832#msg118832

[fbeye]

I also wanted to mention;

OPNSense LAN IP - 172.16.2.1, which connects to 172.16.2.2 - Interface IP on SG350XG which hosts several networks.
The Network in question is 192.168.1.180:8080.
On the OPNSense, I do have a static route 192.168.1.0/24 172.16.2.2 so this network and specific IP can indeed be found.
Not sure if there was something extra in the guide I would need to do that was not mentioned being the server being hosted is not on the OPNSense firewall or using 172.17.2.1. But, OPNSense can ping the 192.168.1.180 so there is a path.. Not sure if that would change the dynamic though...

[fbeye]

Topology if it helps

[fbeye]

Not sure how to "close" this thread. It clearly does not work so I deleted the plugins and moved on. I'm good.

[fbeye]

Quick afterthought... Could Zenarmor by any chance have blocked the incoming connections by dns or anything?

[cookiemonster]

I used nginx in the past on OPN as reverse proxy and had no problems I couldn't solve. Key is tracing the requests along the path, and for that we need logs. I use only VMs or bsd jails, not docker, so couldn't help with that.
I couldn't see the problem in this thread btw.

[fbeye]

Well maybe I will try it all again. Ugh, I must have missed something but what I did was take notes after each thing I did as to know what I did or did not do as I followed the guide.

[cookiemonster]

what is the problem you experience though?

[fbeye]

Well like I said I did the guide every step but of course putting in my own information. I am trying something simple first, just access to the bittorrent GUI on 192.168.2.181:8080, just for fun and to experiment. I have a domain qbitmain.x.duckdns.org which is "registered" to my WAN [OPnsense firewall] x.x.x.182.
I can "see" it trying to connect in the OPNS GUI and is in green, which I assume "passes". But no matter what browser I use it just times out trying to connect.
My ONLY thoughts were.....

OPNSense is on the WAN IP and it's LAN is 172.16.2.1 which servers no purpose but to link to 172.16.2.2 which is the Cisco SG350XG.. 192.168.2.181 resides on that v[SG350XG] Switch.
I have no NAT rule or anything at all for 172.16.2.0 Network, except for static routes pointing where to find the networks... I do not see this as an issue because in the OPNSense GUI I can ping 192.168.2.181 so really there should be nothing blocking it. The SG350XG has no ACL's, no firewall. It just serves as a switch/dhcp server.

So essentially, everything appears as it is connecting but does not. I mentioned zenarmor cause looking at that log it even shows the incoming dns request for qbitmain.x.duckdns.org.
I will redo it all right now etc and see what we come up with.

[fbeye]

Firewall rule shows it trying to connect and PASS but it is not being directed it seems.

[Monviech (Cedrik)]

Did you do any packet captures yet to see where the packet flow stops?

Try to follow the steps here, its for caddy but it applies universally to any reverse proxy. Troubleshoot with tcpdump and curl to get an idea what happens and where.

https://docs.opnsense.org/manual/how-tos/caddy.html#faq

[fbeye]

Other than follow the steps in verifying I put in correct direction firewall rules and ports and http/https stuff, I have no idea how to perform a packet capture or tcpdump

[Monviech (Cedrik)]

Go on the VM you try to connect to vis ssh, install a pkg that contains tcpdump (e.g. apt install tcpdump), and do

tcpdump -i interfacename port XXX

So if your interface is igb0 and the port your application listens on is 8443

tcpdump -i igb0 port 8443

Tcpdump can be used on pretty much all unix like OS.

[fbeye]

Interesting..

fbeye@omv181:~$ sudo tcpdump -i br0 port 8080
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
0 packets captured
6 packets received by filter
0 packets dropped by kernel

This machine is running OMV, OpenMediaVault and Qbit is in a docker container, but still using the OMV IP, 192.168.2.181 and is using Port 8080.

So I have no idea if br0 is a legit Interface as in OMV I have it set as a Bridge network to my physical Interface.
It has to be the right Interface cause when I remove port, it is ENDLESSSS scrolling of data. When I put Port 8080 back and let it sit for 2 minutes as I try to connect and stop it, after I view the .pcap file, it is empty. But the file when no port specified is a huge file. So it seems Port 8080 is not being sent through/pushed through the OPNSense?

[Monviech (Cedrik)]

If the OPNsense sends a request to e.g. 172.16.2.100:8080 and you do not see it in tcpdump at all then nothing gets to your host.

If its a linux you can also do

tcpdump -i any port 8080

that captures all existing interfaces.