[TUTORIAL] Nginx as simple reverse proxy with web application firewall and SSL

[michaelgo]

thank you so much for this simple and straightforward guide

[kimtaro]

Hello,
I followed this tutorial to make NGINX as simple reverse proxy but it's doesn't work for me.
When I want to access to my server from outside it doesn't display anything.
I also try to connect my home assistant application but I had this issue :

"Server Error
Sorry, but something went wrong on our side.
There is nothing you can do except wanting until we fix the issue."
Can you help me please?
Thanks you.

[newbee]

reserved for Web application firewall configuration

bump

thanks for the guide

[kimtaro]

Someone can help me please??

[Fright]

@kimtaro

Sorry, but something went wrong on our side

server error error_page
can you please start a new topic and share your config and nginx error logs?

[wbk]

Hi, thanks for the tutorial!

This is my first post on the forum, after reading through quite a few (archived) posts related to nginx and trying to match configuration snippets from conf files on the net with the options in the web GUI. I hope it is ok to post it here as a reply, instead of opening a new topic. Excuses for the length.

Is there a lot that I can do wrong when following the tutorial? I spent a couple of nights on it now, including removal and reinstallation of the nginx plugin and removing/redoing the configuration to the letter and with variations, with reboots in between for good measure. I can't seem to get it to work.

My network is quite simple (for my first post I also drew my first diagram at asciiflow.com, excuse my drawing capabilities) :

┌─────────────────┐
│                 │
│  GPON box       │
│                 │
└───┬─────────────┘
    │
    │
    │
    │
    │ IPv4 + IPv6
┌───┴─────────────┐
│                 │
│ OPNsense        │
│192.168.1.1      │
└───┬─────────────┘
    │
    │
    │
    │
┌───┴────────────┐
│switch          │
└───┬────────────┘
    │
    │
    │
    │
┌───┴────────────┐
│switch          │
└─────┬──────────┴────┐
      │               │
      │               │
      │               │
      │               │
┌─────┼───────────────┼───────────┐
│     │               │           │
│  ┌──┴─────┐     ┌───┴────┐      │
│  │YUNO1   │     │ YUNO2  │      │
│  │ .30    │     │  .31   │      │
│  └────────┘     └────────┘      │
│  PROXMOX                        │
└─────────────────────────────────┘



Without nginx as reverse proxy I can get one of the Yunohosts on the net via my ISP IPv4. My goal is to at least have the websites of multiple Yunohosts available at my public IP, hence nginx.

Steps so far:

• added upstream server
  
• added upstream
• added location
  
• added HTTP server
  
• added pass rule on WAN interface

I have installed the ACME plugin, but have not yet used it (my domainprovider is not supported by the DNS option)

I started out doing that for multiple servers and sites at once, now only for one.

Observations:

• accessing the website via my phone via public internet on the WAN interface shows pass log records in the live view
  
• no logging from the same source IP (phone) appears in nginx access log
• no logging appears in the nginx error log
  
• no logging appears at the upstream server side in its server logs
• there are records in the nginx log for this upstream, with 502 (earlier) and 302 (later on), from internal hosts as well as from external hosts, which confuses me
  

Some notes:

• IPv6 + DHCP is configured on OPNsense, but clients do not seem to get a lease that is available for rules in OPNsense (there is a local link address and a global address (perhaps SLAAC, it resembles the MAC address). The global addresses from clients are in my /48 subnet, but not in a range that OPNsense recognizes.
  
• The yunohosts have IPv6 configured as well as IPv4
  
• Yunohost is quite strict in using TLS and redirecting to a captive portal
• The tutorial states, under 2.2 upstream, 'leave the rest as is if you don't use HTTPS directly on your upstream server. Does that mean: a) when I use 443 instead of 80 under 2.1, upstream server, or b) when I want to use HTTPS between nginx on the firewall and the webserver on the actual upstream server, instead of having plain HTTP in the LAN?

Thanks for bearing with me so far. I have no clue where to continue troubleshooting, and would appreciate any help!

[fabian]

When your backend server is communicating via HTTPS, you have to enable it in the upstream. Otherwise, nginx will connect via HTTP to that port.

[wbk]

When your backend server is communicating via HTTPS, you have to enable it in the upstream. Otherwise, nginx will connect via HTTP to that port.

Thanks for your quick reply!

TL;DR: I ticked 'Enable TLS', and experimented more after that did not verifiably solve the problem (not reachable, no log entries). I experimented more, also without result.

Long version:
I have been experimenting with ticking the 'Enable TLS (HTTPS)'-box under upstream, but that was without knowing whether it was needed or not. Now I won't turn it of, and am experimenting with and without 'TLS: Servername override' (giving it the identical name as the domain name for that actual upstream server vs empty).

In the 'Upstream Server'-config, I had one line for the upstream server, with its internal IP and 80 as port as per the  tutorial. I added a second 'upstream server' line, for port 443, and linked it in the upstream together with the upstream server for port 80 (after two separate upstreams did not work)

In the 'Location' config there's a tick box 'Force HTTPS'. Is the result of that setting that nginx will switch the connection from 80 to 443, instead of relying on the upstream server to do the redirect from 80 to 443? For the time being, I leave that box unchecked, as it is mentioned nowhere. The upstream server will switch to HTTPS / 443 anyway and most clients will connect via HTTPS as well (apart from Letsencrypt for the upstream server's ACME client).
I duplicated the 'Location' line as well, to be able to use both upstream configurations, so I got 2 'Location' lines, but then nginx would not restart, complaining about duplicate locations. That implies one of my upstreams is superfluous, so I reverted that and the second upstream as mentioned above.

In the 'HTTP Server' config I have one line, with one server name (the domain name that points to the external IPv4, and that I mention in the override in upstream when not leaving it empty).
If the actual upstream server got its own TLS certificate, is it necessary to make it available in OPNsense so I can pick it here under 'TLS certificate'?

When comparing this to any tutorial I am sure I making things needlessly complex, but things just don't get rolling with the simple setup (neither do they now...)

The crazy thing is, totally unrelated lines still turn up in the nginx access log, such as 302's on GETs to a deeplink on the upstream server's domain, or POSTs to a deeplink to non existing deeplink on the external IP (scanning for vulnarabilities, I guess, but I don't understand why they show up in the access log when there was no configuration for them in nginx).

In the mean time, the actual upstream server is accessible from the LAN using a custom hosts entry on my workstation,
- edit - I mean, at least the server is accessible in the LAN, so it is not that the upstream server itself has a problem.

[wbk]

There is progress! Anyway, there are some errors in the nginx error log that relate to the domain name of the upstream (gialinh.nl) :-)

When enabling TLS in the upstream, does that do something to the readability of the logging? I got log records from connections on the LAN-side with hostnames not known to me, which gives me an uneasy feeling:

Code Select Expand

*47 upstream SSL certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: 192.168.1.122, server: gialinh.nl, request: "HEAD / HTTP/1.1", upstream: "https://192.168.1.30:443/", host: "tgizmkzpcubin"
*47 no live upstreams while connecting to upstream, client: 192.168.1.122, server: gialinh.nl, request: "HEAD / HTTP/1.1", upstream: "https://upstream670c050f850943b19359b3fa719f5a7b/", host: "tgizmkzpcubin"
*51 no live upstreams while connecting to upstream, client: 192.168.1.122, server: gialinh.nl, request: "HEAD / HTTP/1.1", upstream: "https://upstream670c050f850943b19359b3fa719f5a7b/", host: "kcqexiggecaupwx"
*53 upstream SSL certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: 192.169.139.161, server: gialinh.nl, request: "GET /services/blog/2014/10/26/apk/ HTTP/1.0", upstream: "https://192.168.1.30:443/services/blog/2014/10/26/apk/", host: "osba.nl", referrer: "http://osba.nl/services/blog/2014/10/26/apk/"


• 
  The hostname of 192.168.1.30 is gialinh, domain is gialinh.nl.
• The certificate of gialinh.nl is valid until December 2021
• There are 2 domains configured on 192.168.1.30 (osba.nl and gialinh.nl), and 6 subdomains. One of the subdomains (www.osba.nl) has an expired certificate, the others have valid certificates.

Thinking about it, my TLS servername override in the upstream would be responsible for some of those symptoms, would it not? It overrides all requests for any of the (sub)domains at the actual upstream server, so that at some point there's a mismatch in the connection chain. I will be removing the TLS servername override from the upstream. I will change the 'HTTP(S) Server' configuration and add the other (sub)domains under 'Server Name'.

[wbk]

OK, seems to help, one way or another. I got my phon's IPv4 in the error log now. and in the access log as well.

Still no website on screen, owing to a bad gateway.

Curious thing: port 80 on the WAN side, via domain gialinh.nl, shows an error message (Server error, sorry, but something went wrong on our side. There is nothing you can do except waiting util we fix the issue | Web Application Protection by OPNsense).

I have two 'Upstream Server'-lines for 192.168.1.30, one for port 80 and one for port 443, both connected to this one upstream record. The upstream record is connected to the single 'HTTP(S) / Location' record, which in turn is connected to a single 'HTTP(S) / HTTP Server' record with in that record 8 (sub)domains.

Logging seems to suggest I'm getting close, this is in the error log:

Code Select Expand

*14 upstream SSL certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: 178.227.96.232, server: gialinh.nl, request: "GET / HTTP/1.1", upstream: "https://192.168.1.30:443/", host: "gialinh.nl"
*14 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 178.227.96.232, server: gialinh.nl, request: "GET / HTTP/1.1", upstream: "https://192.168.1.30:80/", host: "gialinh.nl"
*14 no live upstreams while connecting to upstream, client: 178.227.96.232, server: gialinh.nl, request: "GET /favicon.ico HTTP/1.1", upstream: "https://upstream670c050f850943b19359b3fa719f5a7b/favicon.ico", host: "gialinh.nl", referrer: "http://gialinh.nl/"

There are some records in the access log as well, from my phone (178.227.96.232) and others, all get HTTP 502.

Any hint where to look?

[wbk]

Still getting closer.

On 'HTTP Server' I ticked 'HTTPS Only', because on closer look I noticed port 80 in some of the error logs for failed handshakes:

Code Select Expand

*5 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 178.227.96.232, server: gialinh.nl, request: "GET / HTTP/1.1", upstream: "https://192.168.1.30:80/", host: "gialinh.nl"

Besides that. in 'Upstream' I disabled TLSv1/TLSv1.1 because as far as I know, Yunohost without modification will only talk 1.2 or newer.

So far, the only requests that made it to the upstream server were lines like:

Code Select Expand

/var/log/nginx# tail -f access.log
192.168.1.1 - - [26/Sep/2021:11:53:50 +0000] "\x16\x03\x01\x00\xBD\x01\x00\x00\xB9\x03\x03\xD8\x95{\xE3\xD6XX\xFA9?\xFCz\xDA\xB9\x8A\x18\xE6G\x5C\x1E\x9A\xC2&\xF9k\xA3\xBF\xB0\xE5j%m\x00\x008\xC0,\xC00\x00\x9F\xCC\xA9\xCC\xA8\xCC\xAA\xC0+\xC0/\x00\x9E\xC0$\xC0(\x00k\xC0#\xC0'\x00g\xC0" 400 166 "-" "-"
(so, this line is _not_ logged by nginx on OPNsense, but on the upstream server).
Once forcing HTTPS, those disappeared. Good riddance, but now not any requests make it to 192.168.1.30 at all, for any of the domains.

In the access log there are no HTTP 502's anymore. Now all requests that make it thus far, are logged with HTTP 302.

On my phone, when trying to reach http://gialinh.nl, I do get redirected to https://gialinh.nl, but then I get a timeout. Since enabling 'HTTPS Only', the IP of my phone does not turn up in the logging anymore (not in error, nor in access, for sure not in upstream logging).

In HTTPS Server I changed 'Access Log Format' to 'Extended', hoping to scrape suggestions from the access log, but it does not seem to be of influence.

I am too far out of my depth to do a systematic troubleshooting and fall back to ticking boxes and reverting them when the situation gets worse. Thank you for reading my autobiography thus far, I am looking forward to any hints that can help me further.

[fabian]

By the log I guess you try to access an http port with https.

[wbk]

Hi Fabian,

Thank you so much for your attention, the frustration is driving me crazy!

By the log I guess you try to access an http port with https.

I noticed the same, that is why I ticked 'HTTPS Only' which I could have written more clearly. It did have an effect, because those log records disappeared (and the "\x16\x03\x01\..." records at the upstream end), but I still do not get through to the upstream servers.

Presuming you do not mind spending more time on my problem: which information can help in troubleshooting, and which way of presenting is most helpful?

[wbk]

Just to be sure: I don't need NAT for port 80 and 443 to get this to work, do I?

• There is a rule to pass WAN traffic to 'this firewall'
• When I try to open the sites from my phone, WAN traffic is being logged in live view
• All firewall rules have logging enabled, also the generated rules
• No records are being added to either nginx access.log or nginx error.log

I tried doing a packet capture on LAN for 192.168.1.30/24, but it started only once without capturing anything. Since that first time, I have not been able to make any capture on LAN for a specific host address: the screen flickers, and returns with the start button. When I don't specify a host address, or include WAN in the selection, packet capture and viewing works as expected.

[Fright]

@wbk
hi
imho
it looks like http\https methods and ports messed up:

(\x00k\xC0#\xC0'\x00g\xC0" 400 166

its 400 response code to http connection attempt on https port

and it is possible that the connection to the upstream is not fully configured:
it is better to set up a separate upstream for each site and specify the name in the TLS: Servername override field (don't forget to enable TLS SNI forwarding in Location settings)
for the debugging, you can try to disable the TLS: Verify Certificate in upstream settings. you can enable it later