Potential issue with OPNsense and linuxserver.io SWAG/Unraid

[diehardbattery]

I want to preface this by saying I realize there are multiple facets to my issue, but r/Unraid mods keep deleting this post for unknown reasons, so I thought I'd try here...

I'm usually pretty good at following guides to set up things I'd like to have/use. But, when something goes wrong, especially for networking, I'm apparently not yet good enough to troubleshoot it properly (but hey, I can plug and unplug an ethernet cable?), so here we are...  I am doing my best to document every step I have taken thus far, so apologies if it's incomplete.

Currently I am stuck with what appears to be a port forwarding situation on my OPNsense router with regard to SWAG. I get a 502 when trying to access the subdomain from either LAN or another WAN (Verizon Wireless from phone). I can hit the SWAG Welcome Page when trying to access the root domain from another WAN (also Verizon Wireless from phone) but not from LAN.  I am trying to figure out the correct sequence for OPNsense. 

What I've tried so far with OPNsense:

• Sequence: WAN interface ANY > WAN ADDRESS > TCP 80/443 > 18001/18443 > (Unraid IP)/32 (1 entry each)
• In other words, I have port forwarding rules for 80 and 443 from ANY, to WAN address, then forward as ports 18443 and 18001 respectively to the Unraid server with a /32 CIDR.
• Enabling or disabling NAT Reflection (hairpin NAT?).
• Filter rule association set to PASS
• For NAT Outbound, I have it set to hybrid due to having rules for my gaming consoles/upnp.

What I've tried so far for Cloudflare:

• Turning proxy on/off for the domain/subdomains. 
  Purging the site cache.
  
• Changing SSL/TLS encryption modes (Full and Full Strict)

What I have done with SWAG:

• Consulted with linuxserver.io SWAG page under troubleshooting 502 errors: port unchanged except for upstream_port and upstream_app to match guacamole container (see conf below)
• Recreated relevent conf parameters in the ApacheGuacamole Docker template (added labels for swag=enable, swag_port and swag_url)

My guacamole.subdomain.conf:

Code: [Select]

## Version 2021/05/18
# make sure that your dns has a cname set for guacamole and that your guacamole container is not using a base url

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name guaca.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    # enable for ldap auth, fill in ldap details in ldap.conf
    #include /config/nginx/ldap.conf;

    # enable for Authelia
    #include /config/nginx/authelia-server.conf;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
        #auth_request /auth;
        #error_page 401 =200 /ldaplogin;

        # enable for Authelia
        #include /config/nginx/authelia-location.conf;

        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app ApacheGuacamole;
        set $upstream_port 7575;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

        proxy_buffering off;
    }
}
The nginx error log says this:

Code: [Select]

[error] 8002#8002: *6 connect() failed (111: Connection refused) while connecting to upstream, client: xxx.xx.xx.xxx, server: guaca.*, request: "GET / HTTP/2.0", upstream: "http://172.18.0.2:7575/", host: "guaca.xxxxxx.io"
Where client is my WAN IP, guaca.* is my subdomain (with CNAME pointed to @), 172. address is the custom docker network.  If the problem lies within SWAG, I'm not sure what is not correct as there are no errors in the logs.  If it is within OPNsense, I'm also not sure about that as it looks correct to me.

Any other combination of port forwards (i.e. reversing the ports, doing 443/80 for everything, etc) gets a 522. I know something is not right, but I am not experienced enough to determine what.  I wish I knew what to do next... hopefully someday I will lol...

[cookiemonster]

As you say, there seem to be multiple parts to the desired setup. If you don't mind me saying, it's still unclear what the desired setup is.
You mention OPN port forward but then there is cloudflare, nginx, unraid, swag, local and public networks. Those are the ones I spot, maybe there are more.
IMHO an approach would be:
1 - read on the basics of networking. The OPN Documentation or any other firewall documentation is a good place to start. You want to know what is routing, switching, DNS resolution on/with firewalls, including reolving and routing public domains in your own infrastructure.
2. make a diagram that connects boxes that then you can identify ip addresses and ports for. Most firewall/network forums will more easily see where you might have a mistake. They tend to talk on that level. Currently nobody can see your setup's topology. Your posts will be likely to have more helpful responses when they a) have a "clear picture" what you have and could do, b) don't get put off by trying to tease bits of information out of you that they would rightly expect to be clearer from the start.
3. with that, you can then ask separate questions if required. ie. "I'm trying to route mydomain.cloudflare.com to my local server on the lan with ip 172.26.27.5 running Apache/nginx, my wan is on ipv4 dynamic, do I put my pass rule in the WAN or LAN interface?"
4. try to make your post more generic. What is swag? Don't assume OPN users are familiar with that or a piece of software/hardware you have in your mix.

[diehardbattery]

Please don't mistake for what I am about to reply as hostility.  I realize now in my OP that I might have come across as a blithering idiot who has no business messing with these things.  I attempted to make that point with some humor but apparently failed miserably.

1 - read on the basics of networking. The OPN Documentation or any other firewall documentation is a good place to start. You want to know what is routing, switching, DNS resolution on/with firewalls, including reolving and routing public domains in your own infrastructure.

I already know the basics of networking.  I could be wrong but I don't think I would have come this far if I had not?

2. make a diagram that connects boxes that then you can identify ip addresses and ports for. Most firewall/network forums will more easily see where you might have a mistake. They tend to talk on that level. Currently nobody can see your setup's topology. Your posts will be likely to have more helpful responses when they a) have a "clear picture" what you have and could do, b) don't get put off by trying to tease bits of information out of you that they would rightly expect to be clearer from the start.

Granted I did not draw an actual diagram.  But, the sequence I posted in OP is not clear enough?  Can you explain this further please?

3. with that, you can then ask separate questions if required. ie. "I'm trying to route mydomain.cloudflare.com to my local server on the lan with ip 172.26.27.5 running Apache/nginx, my wan is on ipv4 dynamic, do I put my pass rule in the WAN or LAN interface?"

I thought that perhaps having everything in one thread would be more efficient.  Strike 2 for me...  Furthermore, how can one reasonably expect that someone who is still trying to learn to have every bit of information in a nice neat little package?

4. try to make your post more generic. What is swag? Don't assume OPN users are familiar with that or a piece of software/hardware you have in your mix.

I believe there have been other posts on SWAG here in the past.  So once again, I must have mistakenly assumed that perhaps at least SOME users would know what this is.  Surely I'm not the only person who has a similar enough scenario to be able to at least draw ideas from? 

It's basically nginx with added features for Unraid such as mods for docker (linux containers), auto configuration of apps based on preconfigured proxy confs (they provide a bunch of samples) which can be used in an Unraid environment.

At the core it might not matter.  Port forwarding from 80/443 on my WAN (AT&T Fiber Gateway) is somehow not working properly to reach my internal Unraid server.  I have already outlined the different options I've tried in the OPNsense firewall Port Forwarding.

The reason I am asking for help is that, based on the nginx error log, nginx IS, in fact, able to see that SOMETHING is incoming.  But it does not know what, and is refusing the connection.  The part I am not sure about is if it is an OPNsense thing or a SWAG thing.

On a more personal note, not everyone learns the same way.  So I apologize for my vagueness and inability to concisely convey what is going on.  I want to learn otherwise I would not have posted at all...

Again, I'm not trying to be hostile, but perhaps venting a bit at my situation. I feel like I am in some parallel universe where I understand everything but when I communicate it's gibberish.

[cookiemonster]

No worries. I _think_ I get a the problem a little.
It seems your port forwarding is fine then: client request -> WAN -> port-forward to (lan ip by local name resolution) -> nginx.
There, nginx is listening on port 443 terminating ssl and forwarding to an upstream on port 7575 where it gets a connection refused.
But you already know that, so I'd focus on that last leg.
I don't know what role cloudflare plays. And I realise now from the post that this has some docker stuff in the mix. So I'm out, it's outside my expertise.
As I said though, it seems the port forward is working, so problem seems downstream from opn. Good luck.

[diehardbattery]

So after some more troubleshooting, I believe I have ruled everything out but the OPNsense downstream that you mentioned.  Now I just want to verify that I have the port forwards setup correctly... I have also verified that ports 80/443 are open from canyouseeme.org.

https://imgur.com/a/E2xEkse

[BoodahsFever]

Why don't you expose a simple docker container on the unraid ip without swag to check if you are doing port forwarding right?

[diehardbattery]

I could do that.  But, if nginx is refusing the connection, wouldnt that mean that the port forwarding is working on at least some level?  If I am understanding correctly, it is reaching nginx.

[BoodahsFever]

It means port forwarding is most likely working fine and you need to check NGINX or the backend you want to proxy. It might help if you look at: firewall->logs->Live View and make sure the firewall rule(s) have logging enabled.
 
Also make sure that NGINX is allowed access to internet the most obvious reason i can think (although farfetched) of is you seeing packets in NGINX and the rule not working is because outbound traffic is not working.

It however is way more likely the Reverse Proxy configuration is not working correctly or the backend refuses the connection and NGINX passes the connection refused on in it's logging which you in your turn then misunderstand. So also check the logs of the app you want to expose.

Most of the time Firewall rules and port forwarding are more simple to setup then a reverse proxy setup IMO.

[diehardbattery]

I  figured it out.  There was an issue in the proxy.conf file.  Typically, if you follow the swag guide for ibracorp, you should be able to do everything by host name as long as everything is in the same custom docker network.  Even though it's true in my case, I have to modify the proxy_pass line in the conf file and use my unraid ip instead.  It works fine then.