OPNsense Forum

English Forums => Web Proxy Filtering and Caching => Topic started by: always_learning on January 16, 2021, 08:58:30 pm

Title: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: always_learning on January 16, 2021, 08:58:30 pm
I recently was successful at setting up a reverse proxy with Nginx by following this guide: https://forum.opnsense.org/index.php?topic=19305.0

The issue I have run into is that I would like to take that setup and link to different local servers and ports for different purposes.

For example:

https://stuff.example.com > 172.16.0.1 : 80
https://stuff.example.com/docs > 172.16.0.1 : 8080
https://stuff.example.com/music > 172.16.0.5 : 80

And so on.

I thought that adding a Location and specifying URL Pattern and Rewrite rules would enable this, however, either that is not the correct method, or I have been doing it wrong.

How can I accomplish this type of setup?  Is it better to simply use multiple subdomains and proxy each individually?
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: fabian on January 18, 2021, 05:24:53 pm
you can just use multiple locations and assign them to the same HTTP Server

/
/docs
/music

Each of them has a different upstream.
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: always_learning on January 18, 2021, 06:12:04 pm
you can just use multiple locations and assign them to the same HTTP Server

/
/docs
/music

Each of them has a different upstream.

Yes, I have done that.  I have two locations set up, each with a different upstream.  I think it is the pattern matching itself that I am getting wrong, somehow.  I am able to successfully connect to the second server from the base URL (i.e. stuff.example.com) if I give it the base location of /.  Otherwise, when I use the URL pattern, I can't get any connection at all.

The attached image "Capture", shows two different locations, one for Nextcloud as /, and one for OnlyOffice as /docs.  Capture2 shows the details of how the OnlyOffice location is set up.  Capture3 shows the URL rewrite I am attempting to use with the OnlyOffice location (I have attempted connecting both with and without a rewrite).

Any advice as to what I am doing wrong?
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: fabian on January 18, 2021, 11:38:37 pm
Just use the default match type. Yours is not a regular expression.

The default is path contains.
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: always_learning on January 19, 2021, 06:40:31 am
Just use the default match type. Yours is not a regular expression.

The default is path contains.

Please forgive me for my ineptitude, but I am somehow messing something up still.

- I have an https server set up to listen on both 80 and 443 and to redirect all traffic to 443.
- The server is watching for incoming traffic for a domain like stuff.example.com
- The server is directing traffic through 2 locations.
   - Location 1 is just the base url of stuff.example.com which is directed to an upstream that connects to and serves a Nextcloud instance from a machine on a private LAN with an IP address like 172.16.0.1
   - Location 2 is only modified by having a /docs at the end.  For example: stuff.example.com/docs, but also has a url rewrite rule with the goal of stripping it of the "/docs" so that it is seen by the upstream as simply the base address of just stuff.example.com/ but rather than port 80, it is sent to port 8080 on the same machine of 172.16.0.1

When I look at all the match types, I do not see any that simply say "path contains" (see attached image).  Also, I am not sure whether or not the URL rewrite rule is actually stripping the /docs off the end of the url, or if I am messing up the configuration in some other way, however, the most success I have had so far was to get a 404 error page to display.

I appreciate the help so far, and thank you in advance for all continued assistance.
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: Fright on January 19, 2021, 08:44:39 am
Quote
goal of stripping it of the "/docs" so that it is seen by the upstream as simply the base address of just stuff.example.com/
i think it should be something like
Code: [Select]
rewrite ^/docs(.*)$ $1 break;just keep in mind that everything is not quite that simple when you play with uri like that
if it might be useful:
https://forum.opnsense.org/index.php?topic=19122.msg87622#msg87622
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: fabian on January 19, 2021, 08:11:43 pm
Wenn der Präfix hinten wieder weg soll, würde ich da eher die upstream url umschreiben und dabei keine rewrite Regel nehmen sondern location mit regex und dann im Path prefix referenzieren.
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: Fright on January 20, 2021, 07:17:36 am
@fabian
could you give an example please?
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: fabian on January 20, 2021, 10:21:42 pm
I had something like this in the past - cannot remember the exact config

location ~ ^/context(?<apppath>.*)$ {
  proxy_pass http://upstream-name:port$apppath;
}
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: Fright on January 21, 2021, 06:57:33 am
understood thanks!
that's nifty!
(want to test this in plugin - how variable will be passed to proxy_pass directive via config\template)

but @always_learning wants to switch upstreams by the "/context"
so it seems to me that this method is not suitable in this case

so in the context of the question (switching upstreams by dir name and cutting out the dir name) it might even be easier like:
Code: [Select]
location /music/ {
    proxy_pass http://UpstreamForMusicSiteUID/;  # trailing slash is the key for dir stripping
}
need to test how it will work with just "/" in "Path Prefix" field.
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: always_learning on January 21, 2021, 07:41:11 am
understood thanks!
that's nifty!
(want to test this in plugin - how variable will be passed to proxy_pass directive via config\template)

but @always_learning wants to switch upstreams by the "/context"
so it seems to me that this method is not suitable in this case

so in the context of the question (switching upstreams by dir name and cutting out the dir name) it might even be easier like:
Code: [Select]
location /music/ {
    proxy_pass http://UpstreamForMusicSiteUID/;  # trailing slash is the key for dir stripping
}
need to test how it will work with just "/" in "Path Prefix" field.

I'm happy to test this idea, but how would I get it to work from the GUI?  Or do I just need to directly edit the configuration text?  If so, where is it located in the os?

EDIT:  I know how to make a location, just not where/how to do the proxy pass.
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: Fright on January 21, 2021, 08:08:55 am
Quote
I'm happy to test this idea, but how would I get it to work from the GUI?
works at first glance: correct conf record for proxy_pass generated (with trailing slash) and request goes to right upstream without dir


Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: Fright on January 21, 2021, 09:06:17 am
Quote
just not where/how to do the proxy pass
OPNsense generates it for you in /usr/local/etc/nginx/nginx.conf based on GUI parameters )
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: always_learning on January 23, 2021, 02:37:21 am
Quote
just not where/how to do the proxy pass
OPNsense generates it for you in /usr/local/etc/nginx/nginx.conf based on GUI parameters )

Is there a secondary location?  I just made the exact location configuration shown in your previous comment, then when it still resulted in a 404 page, I opened and thoroughly searched through the configuration document multiple times, and while I can find the basic server configuration, the location configuration for /docs/, is nowhere to be found.
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: Fright on January 23, 2021, 06:21:34 am
Quote
Is there a secondary location?
no, this is the only location
Quote
while I can find the basic server configuration, the location configuration for /docs/, is nowhere to be found.
hm. sorry, but did you "apply" after making changes? )
inside the server configuration block, after the directives for the logs and error pages paths, the locations configuration blocks should begin: first built-in, and then added by you through the GUI
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: always_learning on January 25, 2021, 03:18:25 am
hm. sorry, but did you "apply" after making changes? )
inside the server configuration block, after the directives for the logs and error pages paths, the locations configuration blocks should begin: first built-in, and then added by you through the GUI

Yes.  I did make sure to apply the settings by clicking the "apply" button on the General Settings tab of the configuration page.  I haven't tried restarting or anything, but when I log out and log back in and then navigate to the nginx plugin settings, the custom location configuration is persistent.
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: Fright on January 25, 2021, 07:25:05 am
sorry, realy dont know what to say. other nginx.conf file is a template file at /usr/local/opnsense/service/templates/opnsense/nginx/ but its small and contains only modules and includes

real nginx config should be in /usr/local/etc/nginx/nginx.conf
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: always_learning on February 02, 2021, 05:02:31 am
sorry, realy dont know what to say. other nginx.conf file is a template file at /usr/local/opnsense/service/templates/opnsense/nginx/ but its small and contains only modules and includes

real nginx config should be in /usr/local/etc/nginx/nginx.conf

I have decided to come back to this to try and figure out a little more.  In the nginx.conf file located in the directory you indicated, it seems that all the locations are generic but are linked to upstream servers that are identified by a generated ID, as shown in the image.  I can't change the proxy_pass variable, because it points to the upstream ID.  Take a look at the two images I have attached to see what I'm talking about.

Thanks!
Title: Re: NGINX: Can I use URL Pattern Matching to reverse proxy to different servers?
Post by: Fright on February 02, 2021, 06:23:33 am
hi
yes. nginx conf looks like it should. you do not need (and you should not, since the file is generated every time you apply the settings) to make changes in the file itself. can make it in GUI.
when you set "/" in "Path prefix field" (look at attached image at https://forum.opnsense.org/index.php?topic=20943.msg98074#msg98074) plugin will generate this directive for you. and you will see that a trailing slash has appeared in the proxy_pass directive