Hello
I have NPM running on an internal VM/Docker and it works fine but wanted to set it up to be on the opnsense appliance instead and seem to be having some serious issues.
My WAN is static IP and I have a duckdns.org attached to it and verified it is directed as such.
My OPNSense LAN Interface is 172.16.2.1 which is of course the GUI access address..
My OPNSense Port 80 and 443 have been changed so that those 2 ports are free and available for NGINX.
I followed this guide to the T obviously putting in the correct code relevant to mys setup.
I am not sure what I am missing.. And either I am not educated enough in log monitoring to see what I am missing or what, but I see no obvious issues... During the setup everything was set and applied without errors.. The certificates were processed and approved etc.
https://forum.opnsense.org/index.php?topic=24778.msg118832#msg118832
I also wanted to mention;
OPNSense LAN IP - 172.16.2.1, which connects to 172.16.2.2 - Interface IP on SG350XG which hosts several networks.
The Network in question is 192.168.1.180:8080.
On the OPNSense, I do have a static route 192.168.1.0/24 172.16.2.2 so this network and specific IP can indeed be found.
Not sure if there was something extra in the guide I would need to do that was not mentioned being the server being hosted is not on the OPNSense firewall or using 172.17.2.1. But, OPNSense can ping the 192.168.1.180 so there is a path.. Not sure if that would change the dynamic though...
Topology if it helps
Not sure how to "close" this thread. It clearly does not work so I deleted the plugins and moved on. I'm good.
Quick afterthought... Could Zenarmor by any chance have blocked the incoming connections by dns or anything?
I used nginx in the past on OPN as reverse proxy and had no problems I couldn't solve. Key is tracing the requests along the path, and for that we need logs. I use only VMs or bsd jails, not docker, so couldn't help with that.
I couldn't see the problem in this thread btw.
Well maybe I will try it all again. Ugh, I must have missed something but what I did was take notes after each thing I did as to know what I did or did not do as I followed the guide.
what is the problem you experience though?
Well like I said I did the guide every step but of course putting in my own information. I am trying something simple first, just access to the bittorrent GUI on 192.168.2.181:8080, just for fun and to experiment. I have a domain qbitmain.x.duckdns.org which is "registered" to my WAN [OPnsense firewall] x.x.x.182.
I can "see" it trying to connect in the OPNS GUI and is in green, which I assume "passes". But no matter what browser I use it just times out trying to connect.
My ONLY thoughts were.....
OPNSense is on the WAN IP and it's LAN is 172.16.2.1 which servers no purpose but to link to 172.16.2.2 which is the Cisco SG350XG.. 192.168.2.181 resides on that v[SG350XG] Switch.
I have no NAT rule or anything at all for 172.16.2.0 Network, except for static routes pointing where to find the networks... I do not see this as an issue because in the OPNSense GUI I can ping 192.168.2.181 so really there should be nothing blocking it. The SG350XG has no ACL's, no firewall. It just serves as a switch/dhcp server.
So essentially, everything appears as it is connecting but does not. I mentioned zenarmor cause looking at that log it even shows the incoming dns request for qbitmain.x.duckdns.org.
I will redo it all right now etc and see what we come up with.
Firewall rule shows it trying to connect and PASS but it is not being directed it seems.
Did you do any packet captures yet to see where the packet flow stops?
Try to follow the steps here, its for caddy but it applies universally to any reverse proxy. Troubleshoot with tcpdump and curl to get an idea what happens and where.
https://docs.opnsense.org/manual/how-tos/caddy.html#faq
Other than follow the steps in verifying I put in correct direction firewall rules and ports and http/https stuff, I have no idea how to perform a packet capture or tcpdump
Go on the VM you try to connect to vis ssh, install a pkg that contains tcpdump (e.g. apt install tcpdump), and do
tcpdump -i interfacename port XXX
So if your interface is igb0 and the port your application listens on is 8443
tcpdump -i igb0 port 8443
Tcpdump can be used on pretty much all unix like OS.
Interesting..
Quotefbeye@omv181:~$ sudo tcpdump -i br0 port 8080
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C
0 packets captured
6 packets received by filter
0 packets dropped by kernel
This machine is running OMV, OpenMediaVault and Qbit is in a docker container, but still using the OMV IP, 192.168.2.181 and is using Port 8080.
So I have no idea if br0 is a legit Interface as in OMV I have it set as a Bridge network to my physical Interface.
It has to be the right Interface cause when I remove port, it is ENDLESSSS scrolling of data. When I put Port 8080 back and let it sit for 2 minutes as I try to connect and stop it, after I view the .pcap file, it is empty. But the file when no port specified is a huge file. So it seems Port 8080 is not being sent through/pushed through the OPNSense?
If the OPNsense sends a request to e.g. 172.16.2.100:8080 and you do not see it in tcpdump at all then nothing gets to your host.
If its a linux you can also do
tcpdump -i any port 8080
that captures all existing interfaces.
Well I have 20 docker containers on there. I have no issues with anything going in or out, only now trying to go through NGINX.
Now 172.16.2.0 has ZERO Rules or NAT or anything, it is simply the link between OPNSENSE and SG350XG. Everything works fine when I do port forward stuff (in terms of not using NGINX but like access the stuff just with port forward).
I can't imagine 172.16.2.0 even needs any port forward or rule as I have static routes to "find" 192.168.1.0 through it
I guess what I am saying is, without utilizing nginx on Opnsens, I can access anything on any network on any of the 6 Networks I have with port forward NAT. So these services are open and running as they should. For some reason my nginx is not relaying to them. Like right now if I set a port forward for 8080 I'll
Access qbit from the internet easy. It's just when I don't port forward and try through nginx , it don't make it.
would you be willing to try haproxy instead? I find it easier to figure out settings as it (the plugin) exposes more options for a reverse proxy and has the "config export" utility that allows to visualise the result of settings in the UI.
Absolutely... I only did the NGINX plugin cause I currently run NGINX in VM/Docker and actually found the OPNS plugin to be 1000% different, so no loyalties to it.
I guess I went with the NGINX as well cause it had that visual step by step, and not assume I know anything, guide to it. But yes I'll install it no problem.
After your will has been broken by HA Proxy try out Caddy, which is very easy to configure and does what you need and has proper documentation (I linked it earlier).
Hello, my name is Matt, I have a serious mental disability.
Installed Caddy, set up the LAN/WAN Rules as suggested, changed GUI to different port etc. applied the LAN/WAN rules as suggested.
Under Caddy;
Domain:
https://
service.host.duckdns.org
443
ACME
Handler:
https://service.host.duckdns.org
Subdomain- None
Reverse Proxy
http://
192.168.2.181
8080
When I try to connect just times out. I connected to my WG and verified is up and running (qbit on 192.168.2.181:8080).
What you did must be all correct though.
What does the caddy debug log say? Post the output when you connect to your domain (set log level to debug).
Quote02T18:01:45 Error caddy "error","ts":"2024-12-02T18:01:45Z","logger":"http.log.error","msg":"dial tcp 192.168.2.181:8080: i/o timeout","request":{"remote_ip":"108.147.172.59","remote_port":"42719","client_ip":"108.147.172.59","proto":"HTTP/1.1","method":"GET","host":"qbitmain.h0use0fm.duckdns.org","uri":"/favicon.ico","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Gpc":["1"],"Priority":["u=6"],"Via":["HTTP/1.1 zphn1amcuc01ism005.wnsnet.attws.com"],"Accept":["image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5"],"Accept-Encoding":["gzip, deflate"],"Connection":["keep-alive"],"Referer":["http://qbitmain.h0use0fm.duckdns.org/"]}},"duration":3.017842969,"status":502,"err_id":"ptig4kk4m","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
2024-12-02T18:01:41 Error caddy "error","ts":"2024-12-02T18:01:41Z","logger":"http.log.error","msg":"dial tcp 192.168.2.181:8080: i/o timeout","request":{"remote_ip":"108.147.172.59","remote_port":"21407","client_ip":"108.147.172.59","proto":"HTTP/1.1","method":"GET","host":"qbitmain.h0use0fm.duckdns.org","uri":"/","headers":{"Accept-Language":["en-US,en;q=0.5"],"Upgrade-Insecure-Requests":["1"],"Sec-Gpc":["1"],"Connection":["keep-alive"],"Priority":["u=0, i"],"Via":["HTTP/1.1 zphn1amcuc01ism005.wnsnet.attws.com"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8"],"Accept-Encoding":["gzip, deflate"]}},"duration":3.005683853,"status":502,"err_id":"5z6rgfpv3","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
Quote2024-12-02T18:01:45 Error caddy error ts:2024-12-02T18:01:45Z logger:http.log.error msg:dial tcp 192.168.2.181:8080: i/o timeout request:{remote_ip:108.147.172.59 remote_port:42719 client_ip:108.147.172.59 proto:HTTP/1.1 method:GET host:qbitmain.h0use0fm.duckdns.org uri:/favicon.ico headers:{User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0] Accept-Language:[en-US,en;q=0.5] Sec-Gpc:[1] Priority:[u=6] Via:[HTTP/1.1 zphn1amcuc01ism005.wnsnet.attws.com] Accept:[image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5] Accept-Encoding:[gzip, deflate] Connection:[keep-alive] Referer:[http://qbitmain.h0use0fm.duckdns.org/]}} duration:3.017842969 status:502 err_id:ptig4kk4m err_trace:reverseproxy.statusError (reverseproxy.go:1269)}
2024-12-02T18:01:41 Error caddy error ts:2024-12-02T18:01:41Z logger:http.log.error msg:dial tcp 192.168.2.181:8080: i/o timeout request:{remote_ip:108.147.172.59 remote_port:21407 client_ip:108.147.172.59 proto:HTTP/1.1 method:GET host:qbitmain.h0use0fm.duckdns.org uri:/ headers:{Accept-Language:[en-US,en;q=0.5] Upgrade-Insecure-Requests:[1] Sec-Gpc:[1] Connection:[keep-alive] Priority:[u=0, i] Via:[HTTP/1.1 zphn1amcuc01ism005.wnsnet.attws.com] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8] Accept-Encoding:[gzip, deflate]}} duration:3.005683853 status:502 err_id:5z6rgfpv3 err_trace:reverseproxy.statusError (reverseproxy.go:1269)
It is like 192.168.2.181:8080 is unreachable, but from OPNSense I can ping that address. Do I maybe need a static route to that IP? Seems I would not I can ping it and also have a static route to that Network, 192.168.2.0.
well the opnsense can not reach it.
Error caddy error ts:2024-12-02T18:01:41Z logger:http.log.error msg:dial tcp 192.168.2.181:8080: i/o timeout
Go to the opnsense ssh shell and try:
curl -vv http://192.168.2.181:8080
Quote@OPNsense:~ # curl -vv http://192.168.2.181:8080
11:31:32.108728 [0-0] * [SETUP] added
11:31:32.109149 [0-0] * Trying 192.168.2.181:8080...
11:31:32.110056 [0-0] * Connected to 192.168.2.181 (192.168.2.181) port 8080
11:31:32.110254 [0-0] * using HTTP/1.x
11:31:32.110495 [0-0] > GET / HTTP/1.1
11:31:32.110495 [0-0] > Host: 192.168.2.181:8080
11:31:32.110495 [0-0] > User-Agent: curl/8.10.1
11:31:32.110495 [0-0] > Accept: */*
11:31:32.110495 [0-0] >
11:31:32.110952 [0-0] * Request completely sent off
11:31:32.114530 [0-0] < HTTP/1.1 200 OK
11:31:32.114646 [0-0] < cache-control: no-store
11:31:32.114711 [0-0] < connection: keep-alive
11:31:32.114787 [0-0] < content-length: 1674
11:31:32.114852 [0-0] < content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; script-src 'self' 'unsafe-inline'; object-src 'none'; form-action 'self'; frame-ancestors 'self';
11:31:32.114917 [0-0] < content-type: text/html
11:31:32.114977 [0-0] < cross-origin-opener-policy: same-origin
11:31:32.115056 [0-0] < date: Mon, 02 Dec 2024 18:31:32 GMT
11:31:32.115115 [0-0] < referrer-policy: same-origin
11:31:32.115175 [0-0] < x-content-type-options: nosniff
11:31:32.115240 [0-0] < x-frame-options: SAMEORIGIN
11:31:32.115320 [0-0] < x-xss-protection: 1; mode=block
11:31:32.115382 [0-0] <
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>qBittorrent Web UI</title>
<link rel="icon" type="image/png" href="images/qbittorrent32.png" />
<link rel="icon" type="image/svg+xml" href="images/qbittorrent-tray.svg" />
<link rel="stylesheet" type="text/css" href="css/login.css?v=1hsus4y" />
<noscript>
<link rel="stylesheet" type="text/css" href="css/noscript.css?v=1hsus4y" />
</noscript>
<script src="scripts/login.js?locale=en&v=1hsus4y"></script>
</head>
<body>
<noscript id="noscript">
<h1>JavaScript Required! You must enable JavaScript for the Web UI to work properly</h1>
</noscript>
<div id="main">
<h1>qBittorrent Web UI</h1>
<div id="logo" class="col">
<img src="images/qbittorrent-tray.svg" alt="qBittorrent logo" />
</div>
<div id="formplace" class="col">
<form id="loginform" method="post" onsubmit="submitLoginForm(event);">
<div class="row">
<label for="username">Username</label><br />
<input type="text" id="username" name="username" autocomplete="username" required />
</div>
<div class="row">
<label for="password">Password</label><br />
<input type="password" id="password" name="password" autocomplete="current-password" required />
</div>
<div class="row">
<input type="submit" id="login" value="Login" />
</div>
</form>
</div>
<div id="error_msg"></div>
</div>
</body>
</html>
11:31:32.115632 [0-0] * Connection #0 to host 192.168.2.181 left intact
I figured it out, and I apologize.
I did not even think about this. In my qbit docker container, I have LAN_ACCESS = 192.168.0.0/16,172.16.2.0/24
It was blocking my remote device, my iPad. So I added that /16 network and it works fine.
Good job figuring it out :D
Not sure if I am getting ahead of myself, but I assume I am able to add my internal [192.168.1.180] Postfix/Dovecot [Ports 25 and 993] to this Caddy? I'd assume I would need to leave WAN Side Ports 25/993 but change the LAN side and then would Crowdsec be able to monitor that via Caddy logs? I saw there was a postfix/Dovecot crowdsec collection but also I know they would have to reside on the firewall/where crowdsec is installed to be utilized.
Crowdsec monitors http access logs that caddy emits.
If its the Layer 4 Proxy there will not be HTTP access logs. So not really.
You could make the Crowdsec setup distributed with collectors and API stuff, but better open a new thread for that or search the forum.
Roger that. Thank you for everythig