Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[braveworm]

Any idea how to setup the VPN_redirect_rule as shown in the P000_001_Schematic?

I wanted to have openvpn and the https sites on the same port but can't figure out the rules and conditions.

[stefan21]

There are a lot of countries or even more and more hotels, where only tcp 80 and 443 are opened in their wifi's or LAN's.

Therefore I'm also highly interested how to configure hyproxy to seperate openvpn traffic from webtraffic. Googling around brings up this: https://community.openvpn.net/openvpn/ticket/1352

" HAProxy is perfectly capable of proxying and load balancing OpenVPN in TCP mode, and to share a single listening port with OpenVPN and HTTPS sites. I have used this configuration for years.

To distinguish OpenVPN traffic from TLS traffic, use the following combination HAProxy ACL conditions in a HAProxy frontend

!{ req.ssl_hello_type 1 } !{ req.len 0 }

To distinguish SSH traffic from TLS traffic, use the following combination of HAProxy ACL conditions in a HAProxy frontend:

!{ req.ssl_hello_type 1 } { req.len 0 }

A HAProxy backend can be used to load balance multiple servers. Use TCP mode.

A barebone example configuration is given here:

​https://gist.github.com/zukka77/a5ddb8d81ef9a82e2ff797e3a578c97e

Furthemore, PROXY protocol is protocol agnostic, and could therefore perfectly well be implemented in OpenVPN (Community version). Read more here:

​https://www.haproxy.com/blog/haproxy/proxy-protocol/

PROXY protocol support in OpenVPN would be very welcome, since it will allow OpenVPN servers to know the real IP addresses of connecting clients."

Further more:

https://gist.github.com/zukka77/a5ddb8d81ef9a82e2ff797e3a578c97e

"frontend ssl
    mode tcp
    bind 0.0.0.0:443 name frontend-ssl
    option tcplog
    log global
    tcp-request inspect-delay 3s
    tcp-request content accept  if  { req.ssl_hello_type 1 }
    use_backend main-ssl        if  { req.ssl_hello_type 1 }
    use_backend ssh             if  !{ req.ssl_hello_type 1 } { payload(0,7) -m bin 5353482d322e30 }
    use_backend openvpn         if  !{ req.ssl_hello_type 1 } !{ req.len 0 }
    use_backend ssh             if  !{ req.ssl_hello_type 1 } { req.len 0 }


backend main-ssl
    mode tcp
    server main-ssl 127.0.0.1:8443

backend openvpn
    mode tcp
    server openvpn-localhost 127.0.0.1:1194

backend ssh
    mode tcp
    server ssh-localhost 127.0.0.1:22
"

Let's try together to figure out how this can be translated in OPNsense haproxy. As pre-requisite a openvpn server is running configured to listen on port 1194 and ready to connect to roadwarriors.

In haproxy:

1. I assume it has to be a condition created including !{ req.ssl_hello_type 1 } !{ req.len 0 }
2. a rule is needed which includes the condition and directs the traffic to the vpn server via the vpn backend.
3. a real openvpn server has to be setup
4. a backend vpn is needed

As the SNI_frontend sends most of its traffic to SSL_backend, has this to be integratet in the SNI_frontend or is a new i.e. VPN_frontend needed?

At this point I need help to step further. How exactly has this to be setup? Where are the gurus to get this on the way? IMVHO a lot of people would appreciate a solution.

regards,
stefan

[stefan21]

Here's another solution:

https://www.reddit.com/r/selfhosted/comments/i0iq4g/guide_haproxy_with_openvpn/

"global
   log /dev/log   local0
   log /dev/log   local1 notice
   tune.ssl.default-dh-param 2048
   chroot /var/lib/haproxy
   stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
   stats timeout 30s
   user haproxy
   group haproxy
   daemon
   ca-base /etc/ssl/certs
   crt-base /etc/ssl/private
   ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
   ssl-default-bind-options no-sslv3

defaults
   log      global
   mode   http
   option   httplog
   option   dontlognull
   timeout connect 5s
   timeout client   50s
   timeout server   50s

frontend http
   bind :80
   mode http
   option http-keep-alive
   option forwardfor
   timeout client 30s
   # Redirect everything to https
   redirect scheme https code 301 if !{ ssl_fc }

frontend TLS_passthrough
   bind :443
   mode tcp
   option tcplog
   tcp-request inspect-delay 5s
   tcp-request content accept if { req_ssl_hello_type 1 } or !{ req_ssl_hello_type 1 }
   # Change this to your domain
   use_backend tcp_to_https if { req_ssl_sni -m end .yourdomain.tld }
   default_backend openvpn
      acl http req.ssl_ver gt 0

backend tcp_to_https
   mode tcp
   timeout connect 30s
   timeout server 30s
   server https 127.0.0.1:8443

frontend https
   bind :8443 ssl crt-list /etc/ssl/haproxy.certlist
   mode http
   option http-keep-alive
   option forwardfor
   timeout client 30s

   acl acl_guacamole hdr_beg(host) -i guacamole
   acl acl_plex hdr_beg(host) -i plex

   use_backend guacamole if acl_guacamole
   use_backend plex if acl_plex

backend openvpn
   mode tcp
   timeout connect 30s
   timeout server 30s
   retries 3
   server openvpn 192.168.100.218:443

backend plex
   mode http
   balance source
   stick-table type ip size 50k expire 30m
   stick on src
   timeout connect 30s
   timeout server 30s
   http-reuse never
   server plex 192.168.100.212:32400

backend guacamole
   mode http
   balance source
   stick-table type ip size 50k expire 30m
   stick on src
   timeout connect 30s
   timeout server 30s
   http-reuse never
   server guacamole 192.168.100.201:8084
"

How to translate this to OPNsense??? I tried for a few days, but it's beyond my knowledge.

[stefan21]

Getting closer while following this: https://forum.opnsense.org/index.php?topic=18538.0

For the https_passthrough I set the Type to SSL / HTTPS (TCP mode). The tcp-request inspect delay is set to 10s. (maybe to high?)

For testing I disabled all servers, backends and frontends from the tutorial. For the moment HAProxy passes through to Openvpn on port 443. I'm able to connect my roadwarrior through HAProxy to openvpn on port 443.

HAProxy config looks like this:

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     50
    tune.ssl.ocsp-update.mindelay 300
    tune.ssl.ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 10
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 10

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend (DISABLED): 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)

# Frontend (DISABLED): 1_HTTP_frontend (Listening on 127.0.0.1:80)

# Frontend (DISABLED): 2_HTTPS_frontend (Listening on 127.0.0.1:443)

# Frontend: https_passthrough ()
frontend https_passthrough
    bind 0.0.0.0:443 name 0.0.0.0:443
    mode tcp

    # logging options
    # ACL: traffic_ssl
    acl acl_6741bff6e05423.95774512 req_ssl_hello_type 1
    # ACL: sni_mydomain_condition
    acl acl_6740b3e1c59208.20735171 req.ssl_sni -i abc.mydomain.de

    # ACTION: request_inspect_delay
    # NOTE: actions with no ACLs/conditions will always match
    tcp-request inspect-delay 10s
    # ACTION: request_content_accept_ssl
    tcp-request content accept if acl_6741bff6e05423.95774512
    # ACTION: myservice_sni
    use_backend VPN_backend if acl_6740b3e1c59208.20735171

# Backend (DISABLED): SERVER_backend ()

# Backend (DISABLED): SSL_backend ()

# Backend (DISABLED): MTA_STS_backend ()

# Backend: VPN_backend ()
backend VPN_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server VPN_server my_ip

Now the next tricky part: how to merge/integrate this config into the 0_SNI_frontend of the tutorial?

[pottproll]

Hi all,
did someone get a jitsi instance behind this setup working?

[Patrick M. Hausen]

did someone get a jitsi instance behind this setup working?

Possibly not helpful, but I would seriously consider Caddy for Jitsi, because it does all the websocket upgrade magic automatically.

[pottproll]

I would seriously consider Caddy for Jitsi, because it does all the websocket upgrade magic automatically.

Thanks for the answer! I didnt use caddy so far and would like to add it to the existing haproxy configuration without any other certificates etc. Its just for small (mainly 1:1) meetings. If its with a few additional conditions/rules in haproxy possible, I would prefer that but otherwise I will have a look at caddy.

[Patrick M. Hausen]

Unfortunately if I am not mistaken you cannot run both in parallel but need to replace HAproxy with Caddy at the current state of both plugins. But you might be able to pull some tricks with inbound NAT port forwarding etc. Only that there probably won't be much help based on experience. I use Caddy and HAproxy but never both on the same system(s).

[joeko]

Is it possible to use haproxy if the opnsense is behind or after a fritzbox router ? I have tried a setup but without success

[pottproll]

Is it possible to use haproxy if the opnsense is behind or after a fritzbox router ? I have tried a setup but without success

Yes its possible and fairly common. Just setup the opnsense in the fritzbox as exposed host.

[joeko]

Is it possible to use haproxy if the opnsense is behind or after a fritzbox router ? I have tried a setup but without success

Yes its possible and fairly common. Just setup the opnsense in the fritzbox as exposed host.

i will try it but opening port 80 and 443 for the opnsense is not enough ?

[Rethink7466]

Hello everyone,
first of all thanks for the tutorial :).

Currently I´m setting up HAProxy according this instruction and have been stopped by chapter 5. The first task of this chapter is to enable the checkbox "Store OCSP response". But unfortunately this checkbox is not available on my opnsense.

I also checked the plugin history. I found only the comment in "version 4.2" -> move OCSP settings from "Service" to "Global" section. But here is also nothing.

Therefore I would like to ask you, is this still mandatory to enable this checkbox. If yes, there can I enable this.

Thanks in advance.


Current system:
OPNsense: 24.7.11_2-amd64
os-haproxy: 4.3_2

[meyergru]

OCSP will be abandoned by mid 2025. I guess the plugin has dropped support for it in advance. Likewise, you should also stop using "OCSP Must Staple" in the ACME certificate settings.

[vafk18]

Hello,

thanks for the excellent tutorial. I got HAProxy up and running. I have a TrueNAS Core with docker apps (PhotoPrism & NextCloud) on ports of TrueNAS. The installation of Photoprism as app is working and with HAProxy enabled when I access it with https://ip:port it opens and with http://ip:port the message it says "Client sent an HTTP request to an HTTPS server." which I assume is right. Now when I try to access the PhotoPrism server from WAN with https://sumdomain.domain.com it tells me "503 Service Unavailable No server is available to handle this request." Because HTTP forwards to the same error message I assume HAProxy is working.

I had this same behaviour with my previous installation of HAProxy on pfSense and this was a reason I switched now to OPNSense. I am now stuck here assuming that the problem is not related to HAProxy but to something else I did not figure out yet. Any idea where start? Also would be thankful to find out how to start the statistic page of HAProxy. Many thanks!

[Rethink7466]

OCSP will be abandoned by mid 2025

Thank you very much :).