[SOLVED] Nginx not recognising upstream servers

[dasuberadmin]

I am trying to get the nginx reverse proxy to work with multiple upstream servers using their own ssl cert. The issue that I'm running into is that one of the upstreams is not used and when a client requests a specific webpage the wrong server is used.

When I look in /usr/local/etc/nginx/nginx.conf I see that the upstream servers are listed and point to the correct servers.

In HTTP(S) -> HTTP Server I have 4 servers all listening on the same IP (my OPNsense has only 1 wan ip) on port 443.

In HTTP(S) -> Location I have 3 locations matching "/" with Match Type "None" and no URL rewriting.
Security rules and Learning mode are enabled.
Custom Security Policies are enabled based on the NAXSI WAF.
Upstream servers is pointing to the correct servers.

In Data Streams -> Stream servers I have 2 entries which both listen on the WAN address on port 443 using the TLS cert that is configured by ACME and pointing to the correct Upstream Servers.
The "Route With" option is currently set to "SNI Upstream Mapping" with the correct Upstream Server and corresponding SNI Upstream Mapping.
For this it doesn't matter if I use SNI Upstream Mapping as "Route with" or "Upstream".

In Data Streams -> SNI based routing I have 2 entries.
Entry one has hostname map sub.domain.tld pointing to the correct upstream server and www.sub.domain.tld pointing to the same upstream server.
Entry two is basically the same but using a different domain and different upstream server.

In Upstream -> Upstream server I have 6 entries (3 backend servers using 443 and 80 for which I only need to configure 2 for the time being). All Upstream servers have priority 1.

In Upstream -> Upstream I have 3 entries for the 3 servers that are in the backend.


See attached nginx.conf for reference

[Fright]

i don't think that it's possible to use the same ip:port for http and stream module. there should be an error in nginx log

[dasuberadmin]

Hi Fright,

I indeed see the error you mention. Am I then correct in assuming I need to remove all the hosts at HTTP server and only use the Stream Servers?

[Fright]

Hi
it depends on your needs. if we are talking about HTTP, then this can work both through a ngx_http_proxy_module (more capabilities at the application level) and via streams (deprived of app-level capabilities but should be a bit faster). but you have to choose one 

[dasuberadmin]

Can you explain the difference a bit? I only switched from normal HTTP servers to SNI based routing because the normal HTTP servers did not proxy the connection correctly to the correct backend host.

[Fright]

did not proxy the connection correctly to the correct backend host

did you look at the backend logs?
I think that HTTP should handle it fine, but maybe it’s just worth adding SNI processing in the Location and Upstream settings. so that the backend receives the correct SNI payload.
you can try to turn "TLS SNI Forwarding" on in Location settings and specify "TLS: Servername override" in Upstream settings

[dasuberadmin]

I have now enabled TLS SNI Forwarding in the advanced Location setting for the hostname that I want to map, I have also checked " Enable TLS (HTTPS)" in the Upstream for the host and set the "TLS: Client Certificate" to the ACME cert of the host i want to route. The " TLS: Servername override" now is set to the external subdomain of the upstream host.

However when I go to the desired subdomain still the incorrect cert and incorrect host is chosen. When I look at the HTTP Access logs I see the connection coming in for the wrong host.

Is there anything else I can try?

[Fright]

The " TLS: Servername override" now is set to the external subdomain

it should be set to the fqdn that the upstream expects (may or may not match the external (requst) address)
can you share your current config?

[dasuberadmin]

The FQDN is set correctly AFAIK. See attached config

[Fright]

i dont see proxy_ssl_* directives at all 
is Enable TLS (HTTPS) enabled on Upstream settings?

[dasuberadmin]

Yes, Enable TLS is enabled.

See screenshot (I have removed the (sub)domain names from the screenshot but they are set)

[Fright]

The config was not applied or an error occurred while applying the config?
there are no corresponding directives in the configuration file

[dasuberadmin]

Ah yes, I see there's an error in the nginx log

It is complaining about a duplicate address and port pair. I think that has to do with the fact that I have multiple HTTP Server entries all listening to the same IP and port? Do I need to remove all but 1 entries and cram all my vhosts into that singular HTTP server entry?

[Fright]

no. you can have multiple http servers on the same ip:port
in the config file you still have http servers and streams on the same ports.
can you delete streams if it true?

[dasuberadmin]

I have already deleted the stream servers but maybe nginx.conf is not updating?