HAproxy SSL offloading

[onzi]

Hello. New to OPNsense.

I've been looking for a solution to a specific problem I have and can't find any related articles or even any of my buddies that use OPN/PFsense can't seem to figure it out.

I have three domains and two webservers. All serving on 443.

I've configured the Real Servers (2 web servers). One is a home assistant VM and the other is a CentOS web server with Apache and two vhosts for the two domains serving different content.

I've configured the 3 Virtual Services, one for each domain with SSL offloading. Certificates are installed in OPNsense and are selected to serve for the appropriate domain on the HAproxy virtual service configuration.

I have also configured 2 Backend Pools. One for the Home Assistant and one for the CentOS/Apache vhost server.

What I get when I put this in production is whichever of the three domains I visit, I seem to get round robin into one of the three websites. If I refresh I get the next real server, refresh again I get the next one. One of three refreshes, I get the correct one 😄 If I disable 2 out of the 3 Virtual Services, and only one is serving on 443, it loads normally and works as expected.

I am sure I'm missing some sort of ACL or Conditional access rule, but I can't find any tutorial with use cases.

I come from Sophos XG where part of the problem was resolved by an option to "Pass host header" in the WAF setup. Doing this allows for two domains to be served by one server and the apache makes the differentiation using the vhost configuration.

Here is the configuration for reference, but for me its easier to remember how to do this in the UI:

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    12
    hard-stop-after             60s
    no strict-limits
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua
cache opnsense-haproxy-cache
    total-max-size 10
    max-age 60
    process-vary off

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats



# Frontend: WAN_to_WEB.dmz
frontend WAN_to_WEB.dmz
    bind domain1.com:443 name domain1.com:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/661313b38035d1.75220027.certlist
    mode tcp
    default_backend WEB.dmz.pool

    # logging options

# Frontend: WAN_to_WEB.dmz
frontend WAN_to_WEB.dmz
    bind domain2.com:443 name domain2.com:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6613293011e907.65009651.certlist
    mode http
    option http-keep-alive
    default_backend WEB.dmz.pool

    # logging options
    option httplog

# Frontend: WAN_to_HASS.dmz
frontend WAN_to_HASS.dmz
    bind domain3.com:443 name domain3.com:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6613a9d25d0984.08290998.certlist
    mode http
    option http-keep-alive
    default_backend HASS.dmz.pool

    # logging options

# Backend: HASS.dmz.pool
backend HASS.dmz.pool
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server HASS.dmz internal_IP_1:443 ssl alpn h2,http/1.1 verify none

# Backend: WEB.dmz.pool
backend WEB.dmz.pool
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    option forwarded
    option forwardfor
    server WEB.dmz internal_IP_2:443 ssl alpn h2,http/1.1 verify none

# statistics are DISABLED

Hope someone can point me in the right direction!

Thanks!

[bunchofreeds]

Hi and welcome to OPNsense, it really is an awesome little router/firewall/Swiss army knife.

First off, I'd follow this tutorial and see how you get on.
Then post in the same thread if you get stuck.

https://forum.opnsense.org/index.php?topic=23339.0

I used to run three sites behind OPNsense using HAproxy quite successfully, so it is doable!
One of them was also Home Assistant, another was Guacamole.
It requires following the tutorial above closely. You'll learn a lot about how HA works with OPNsense.

Something else you might want to investigate is running Cloudflared tunnels to each of your web services.
I do this now, it's extremely easy to setup, no more certs plus you get Cloudflare WAF and access challenges.
All within their free stuff

But HAproxy is also good

Edit: Have you moved your OPNsense web front end away from 443? I seem to remember this was one of those changes needed that might cause weird stuff like you're describing.