Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[huuich]

@cookiemonster Thank for your comment about back to original purpose, I'll create a new topic relate my questions. Thanks.

@Bunch: I'm very thankful to you for your help. Best Regards!

[The_Dave]

I have a very serious problem with this haproxy config since I updated to 22.1.3. Suddenly haproxy didn't start anymore. On further investigation trying to start haproxy through the commandline showed that suddenly the ipadresses for the frontend cannot be bound anymore:

Code Select Expand

root@OPNsense:/home/David # /usr/local/etc/rc.d/haproxy start
Starting haproxy.
[ALERT]    (21351) : Starting frontend 1_HTTP_frontend: cannot bind socket (Can't assign requested address) [192.168.64.1:80]
[ALERT]    (21351) : Starting frontend 1_HTTPS_frontend: cannot bind socket (Can't assign requested address) [192.168.64.1:443]
[ALERT]    (21351) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy

Any help on how to fix this would be really appreciated

[Bunch]

It seems that it is the same issue as This thread
I have the same issue after update and reboot.
For temporary fix, edit the VIP, save without any changes, then apply.
You will able to start HAProxy again.

[The_Dave]

For temporary fix, edit the VIP, save without any changes, then apply.

Thank you this actually works for now, I hope there will be a proper solution or fix for this soon though

[balrog]

I had the problem after updating to OPNsense 22.1.3 that the HAProxy service did not start anymore. I was able to solve the problem by editing the Virtual IP and saving it again without adjusting it. After that the service could be started again.

[Morta]

Code Select Expand


2022/03/20 15:00:53 [error] 1124599#1124599: *22208 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.1, server: sync.xxx.ch, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/run/uwsgi/mozilla-firefox-sync-server.sock:", host: "sync.xxx.ch"


I got a 502 Gateway error on this site with nginx.

Where is the option of keep-alive for the backends? Or anybody knows how to fix this error?

[Bunch]

Check and change Services: HAProxy: Settings: Default Parameters
Check Tuning Options of your frontends rather you overwrite the setting too.
If it doesn't fix the issue, then it is Nginx setting problem
Check this link too for Nginx setting

[Morta]

Thanks for reply bunch!

The link is for the settings with a reverse proxy with nginx.

I will look at the tunables of Haproxy.

I found this link

https://serveanswer.com/questions/upstream-prematurely-closed-connection-while-reading-upstream-large-files

Also possible that is a connection error to the SQLite database.

I will give a try at home.

[Morta]

I try to add full IPv6 support to my Haproxy configuration.
Now I have a problem with IPv6 localhost or loopback address

I did a real server ssl_server_ipv6 with IPv6 ::1




I got following error when I add ssl_server_ipv6 to SSL_backend


How I can fix it or do a workaround?

[Morta]

Code Select Expand


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 1000s
    timeout connect 1000s
    timeout server 1000s
    retries 3
    default-server init-addr libc,last
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: SNI_frontend (Listening o)
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    bind :::80 name :::80
    bind :::443 name :::443
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 1000s

    # logging options

# Frontend: HTTP_frontend (Listening 127.0.0.1:80)
frontend HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    bind [::1]:80 name [::1]:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 1000s

    # logging options
    # ACL: NoSSL_condition
    acl acl_621d0b77c74989.24704837 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_621d0b77c74989.24704837

# Frontend: HTTPS_frontend (Listinging on 127.0.0.1:443)
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    bind [::1]:443 name [::1]:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/621d11c7cad951.61400293.certlist
    mode http
    option http-keep-alive
    default_backend WEBSERVER_backend
    option forwardfor
    # tuning options
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/621d0c7054ddb7.46420139.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: WEBSERVER_backend ()
backend WEBSERVER_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server WEBSERVER_server 192.168.1.100:80 send-proxy-v2 check-send-proxy
    server WEBSERVER_server_ipv6 2a02:XXX:XXX::2000:80 send-proxy-v2 check-send-proxy

# Backend: NAS_backend ()
backend NAS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server NAS_server 192.168.1.118:80
    server NAS_server_ipv6 2a02:XXX:XXX::1000:80

# Backend: WEBSERVER_SSL_backend ()
backend WEBSERVER_SSL_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 1000s
    timeout server 1000s
    # WARNING: pass through options below this line
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-reuse safe
    server WEBSERVER_server_ssl 192.168.1.100:443
    server WEBSERVER_server_ssl_ipv6 2a02:XXX:XXX::2000:443


It's this a correct and possible configuration?

[Bunch]

I don't think you need to create another ipv6 real server, as long as it is the same sever in ipv4
You only need to add :::443 and :::80 to frontend listener (in frontend, [::]:80 is the same as :::80, in case you confused with the syntax)
That will be ipv6 to 4 setup.

If you add 2 real server to the same backend, you are load balancing them.

[Morta]

I don't think you need to create another ipv6 real server, as long as it is the same sever in ipv4
You only need to add :::443 and :::80 to frontend listener (in frontend, [::]:80 is the same as :::80, in case you confused with the syntax)
That will be ipv6 to 4 setup.

If you add 2 real server to the same backend, you are load balancing them.

Thanks for the Input.
Real IPv6 or Dual Stack support of HAproxy would be nice but so I have a fallback opportunity if IPv4 or IPv6 are out of service by me or the ISP! 

[os_admin]

I have a very serious problem with this haproxy config since I updated to 22.1.3. Suddenly haproxy didn't start anymore. On further investigation trying to start haproxy through the commandline showed that suddenly the ipadresses for the frontend cannot be bound anymore:

This behaviour seems to be gone since 22.1.4_1. No guarantee.

[os_admin]

Just tested it out myself. Basic Auth is so easy to set up that I am not really willing to cover it in this guide.
First create the user(s) in HAProxy. Then in the relevant backends activate basic auth and select the user(s).

Thanks for this tutorial. It saved my ass. I learned a lot about OPNsense and HAProxy. At last I enabled basic auth. on one of my backends. Anything was fine before, but after activating it I can't no longer login into the service web frontend itself. If I access the frontend browser asked for the basic auth, after that I see the login screen of the service, but after put in the service credentials the FE refresh and shows the login screen again.
If I disable basic auth, the service FE works as expected.

What do I miss? Maybe someone out there has a hint...

[Bunch]

Strange,  have just tested one of my backend
Create user->Enable Basic Auth and select Allowed users in backend
It works as expected.

Edit: Just tested a bit deeper, for some pages like unifi controller, it will always redirect to wrong page
For some pages like opnsense web UI, in chrome(PC), it will keep prompting for auth, but in firefox(PC & mobile), everything works normally

BTW, nothing more can be done in haproxy too, as some site in some browser works normally. Thus, the problem is due to webserver and browser.