Reverse proxy setup and firewall rules (HAproxy or nginx)

[0xp]

Hi,

My first post so please be patient with me.

Background;
I've been using OPNsense for a few months at home and have found it feature rich and fits my needs perfectly. The existing setup is running OPNsense 20.1.2-amd64. I've configured an OpenVPN server on it so I can access my network remotely and securely. I've followed the tutorial to set up a NIC on a separate subdomain for a guest network that routes only to the internet and not to my LAN. Setup port forwarding but now removed this in preparation for reverse proxy.

What I'm trying to do now;
Previously I port forward to my servers but now I'd like to run reverse proxy as I'm trying to run more than two servers and think this would be the more appropriate route to take.

I'd like to set this up;

WAN IP: 12.3.45.67 (static)

DNS:
test1.example.com --> 12.3.45.67
test2.example.com --> 12.3.45.67
test3.example.com --> 12.3.45.67
test4.example.com --> 12.3.45.67
test5.example.com --> 12.3.45.67
test6.example.com --> 12.3.45.67

OPNsense: (setup to use 192.168.100.254:8008 on LAN) (single wan interface 12.3.45.67)
test1.example.com --> 192.168.100.11:443
test2.example.com --> 192.168.100.12:80
test3.example.com --> 192.168.100.13:80
test4.example.com --> 192.168.100.14:21
test5.example.com --> 192.168.100.15:80
test6.example.com --> 192.168.100.16:443


I've followed;
https://wiki.opnsense.org/manual/how-tos/haproxy.html
checked;
https://forum.opnsense.org/index.php?topic=15181
https://forum.opnsense.org/index.php?topic=16253

Questions;
Should I be using nginx or HAProxy? (I don't wish to offload the SSL.)
Is there a step that I'm missing? Is there an idiots guide available?

Overview (Is this correct?);
Login to provider and set DNS records.
Install plugin nginx or HA proxy
Configure reverse proxy (I see this varies on which I use I've so far unsuccessfully had a go with HAproxy)
Configure firewall to point to reverse proxy (is this This Firewall or do I specify the LAN IP of the firewall?)
All should work.

I'm going to keep trying with this and will post more up as I progress any help or pointers would be appreciated.

Thanks

[fabian]

Questions;
Should I be using nginx or HAProxy? (I don't wish to offload the SSL.)
Is there a step that I'm missing? Is there an idiots guide available?

In your case both work equaly but in nginx you cannot mix strams with HTTP so you will have to have all entries either with opening the connection (HTTP) or without (Stream Server with upstream map for the different upstreams).
For nginx, there is a guide: https://docs.opnsense.org/manual/how-tos/nginx_streams.html
You can find an overview about all official HTTP and TCP load balancing plugins likely here: https://docs.opnsense.org/plugins.html#web

Overview (Is this correct?);
Login to provider and set DNS records.
Install plugin nginx or HA proxy
Configure reverse proxy (I see this varies on which I use I've so far unsuccessfully had a go with HAproxy)
Configure firewall to point to reverse proxy (is this This Firewall or do I specify the LAN IP of the firewall?)

The first three steps are ok.
Last Step: In theory you only need to open the ports in the firewall for HTTPS (TCP/443) to the firewall itself on WAN to allow external access but I would allow to access the load balancer from everywhere, I would suggest a quick floating rule. Do not use and Port forward rule and make sure the same port is not used by the web interface.

[0xp]

Hi,

Thanks for your quick reply, I didn't get time to try much over the weekend but still sadly no success think I'm being a bit useless with this, it certainly takes a little more than just enabling NAT.

In your case both work equaly but in nginx you cannot mix strams with HTTP

Thanks for the heads up, I wouldn't be averse to using different ports for some of the servers as one of the things running on server 'test5' defaults to 8081. However as that's the case I'll have another go with HAProxy. Thanks also for the links to the guides.

Last Step: In theory you only need to open the ports in the firewall for HTTPS (TCP/443) to the firewall itself on WAN to allow external access but I would allow to access the load balancer from everywhere, I would suggest a quick floating rule. Do not use and Port forward rule and make sure the same port is not used by the web interface.

I try creating rule to allow in Firewall>Rules>WAN IPv4 TCP/UDP Source * Port * Destination This Firewall Port 443 (HTTPS) Gateway * Schedule * and same for port 80 each time its blocked. I try to access from my phone and watch the live view in firewall and see that its blocking with 'Default Deny rule'. Will have ago with floating rule.

I've had another look in the forums and read these, trying to get as much info as possible as but I can't work out where I'm going wrong;
https://forum.opnsense.org/index.php?topic=7823.0
https://forum.opnsense.org/index.php?topic=3143.0


Starting from the beginning again;

Pre) Disable and delete all/any NAT to avoid any complications, I had NAT working to two of the servers on different ports so I know the servers are working, and I know that the DNS is working.

1) Logged into my domain registrar / DNS provider added the appropriate subdomain A records and checked these had propagated.

2) Logged into OPNSense (192.168.100.254:8008)

3) Installed plugin, System>Firmware>Plugins>os-haproxy (installed)

4) Begin setup of HAProxy, Services>HAProxy>Settings

4a) Real servers,
left Enabled ticked
entered name that made sense to me and description e.g. srv_test1_example_com
entered LAN IP in FQDN or IP
entered port in Port
left Mode active [default]
>did this for other servers on LAN<

4b) Created one Backend Pool,
left Enabled ticked
entered name that made sense to me and description e.g. ha_bep_example_com
left Mode as HTTP (layer7) [default]
left Balancing Algorithm as Source-IP Hash [default]
entered servers into Servers box - these auto-completed
>no other settings changed<

4c) Created one Public Services
left Enabled ticked
entered name that made sense to me and description e.g. ha_ps_example_com
set listen addresses, localhost:80 localhost:443 localhost:21
left Type as HTTP / HTTPS (SSL offloading) [default]   -  Wasn't sure on this one so left as default.
set Backend Pool to  ha_bep_example_com
>no other settings changed<

4d) Created conditions,
entered name that made sense to me and description e.g. ha_cond_test1_example_com
set Condition type to Host matches   -   I wasn't sure if this should be Path starts with?
set Host String to  test1.example.com
>did this for other servers on LAN<

4e) Created rules
entered name that made sense to me and description e.g. ha_rule_test1_example_com
set Test type to IF [default]
set Select conditions to appropriate condition on drop down list i.e. ha_cond_test1_example_com
set Logical operator for conditions to none
set Execute function to http-request allow
>did this for other servers on LAN<

4f) Added rules to Public services
Advanced Settings>Rules>Select rules
auto-completed each of the rules

5) Firewall, as per your recommendation created floating rule
set Action to Pass
set Interface to LAN & WAN
set Direction to in
set TCP/IP Version to IPV4
set Protocol to TCP/UDP
set Destination to This Firewall
set port to any - any              -        I would probably lock this down to just the ports I need later by creating extra identical floating rules, just for the ports required.
entered name that made sense to me and description e.g. flfw_ha_allow_example_com


Still doesn't work, any ideas what I'm missing?


I did read;

This took me quite some time, but I have figured it out.
I simply overlooked several times the significance of the following statement in the docs:

If you configure a port that is already in use, the configuration test will be successful but the start of HAProxy will fail silently. Please ensure that the used port is free - especially if the number conflicts with the web configuration of OPNsense.

By setting up the port for the opnsense web interface to something else then 443, the issue was resolved.
Thanks for those who helped out.

Would that be affected the floating firewall rule, I.e. would HA silently crash or is that only if something else is running on a port required by HA? I'm running OpenVPN but that's on a non default high numbered port?

Thanks again for any help.

[stefanpf]

Hi,

you have to bind the Public Service to your
WAN address likewise
e.g.

Code: [Select]

192.168.0.254:443
80.11.12.13:443
If you are have no static IP at your WAN Interface (e.g. when using PPPOE), this can make Problems because the IP isn't available while haproxy ist starting the First time.

In this case you only use a local binding to the internal firewall address

Code: [Select]

192.168.0.254:443and add a nat portforward

Code: [Select]

WAN address 443 -> 192.168.0.254:443
And as you mentioned:
Change the https Port of the web ui to 444 for example and activate "Disable web GUI redirect rule"

[0xp]

Hi, That helped, I have a static IP didn't realise I needed to put the public IP in the;
Services>HAProxy>Settings>Virtual Public Service>Listen Addresses (add public IP & Port e.g. 12.3.45.67:80)
seems obvious now, damn.

I've still got something messed up as I'm now getting a 503 Service Unavailable error when I connect externally. The log from HAProxy shows;
2020-03-30T08:09:45 haproxy[]: Proxy ha_ps_example_com started.
2020-03-30T08:09:45 haproxy[]: Proxy ha_bep_example_com started.
2020-03-30T08:10:21 haproxy[]: Connect from *********:50433 to 12.3.45.67:80 (ha_ps_example_com/HTTP)

At this point I guess from the log it should have worked but I get the 503 error, which I figure is returned by HAProxy saying so server is available to handle the request, so I've got a break somewhere within HAProxy to my server on the LAN.

In this case you only use a local binding to the internal firewall address

Code: [Select]

192.168.0.254:443and add a nat portforward

Code: [Select]

WAN address 443 -> 192.168.0.254:443

I take it I don't need to do this as HAProxy is seeing traffic coming from the WAN as seen by the log.


I think the problem is that I've not set some option correctly and so traffic from the WAN connect to HAProxy but HAProxy fails to hand the traffic to the server. Any ideas what I need to try to resolve this?


As always thanks for your help

[davide]

Here same problem.
It seems that the plugin don't write conditions and rules inside haproxy cofiguration file (/usr/local/etc/haproxy.conf) :-(

Maybe a bug with the new opnsense 20 and plugin?

Any suggestion?

[davide]

Found the solution: you need to add the rule under Public Server

[acdek]

I am having a lot of the same problems,

@davide, or anywone else ... would you mind posting your working config?