Messages

Hoping someone can point me in the right direction. I've setup according to this guide and anything I DO want to offload is working perfectly. But I also have a service I do NOT want offloading and instead to just passthrough haproxy to it's own reverse proxy (nginx). But I keep getting the cert for the working offloaded service. 
I did originally put both domains into the 1 map file, but you'll notice they are now in 2. I have no issue reverting to 1 if that's how it works, but I had the same result. 
When trying the domain not working debug log shows

2025-03-13T15:37:07-06:00	Informational	haproxy	Connect from 123.123.123.123:35560 to 75.158.105.237:443 (1_HTTPS_Frontend/HTTP)
2025-03-13T15:37:07-06:00	Informational	haproxy	123.123.123.123:35488 [13/Mar/2025:15:37:06.986] 0_SNI_frontend SSL_backend/SSL_SERVER 1/0/172 3288 -- 7/4/3/3/0 0/0
2025-03-13T15:37:07-06:00	Informational	haproxy	123.123.123.123:35488 [13/Mar/2025:15:37:06.987] 1_HTTPS_Frontend/127.4.4.3:443: SSL handshake failure
2025-03-13T15:37:06-06:00	Informational	haproxy	123.123.123.123:35372 [13/Mar/2025:15:37:06.576] 0_SNI_frontend SSL_backend/SSL_SERVER 1/0/223 396 -- 5/3/2/2/0 0/0
2025-03-13T15:37:06-06:00	Informational	haproxy	123.123.123.123:35372 [13/Mar/2025:15:37:06.577] 1_HTTPS_Frontend/127.4.4.3:443: SSL handshake failure (error:0A000416:SSL routines::sslv3 alert certificate unknown)
2025-03-13T15:37:06-06:00	Informational	haproxy	123.123.123.123:35328 [13/Mar/2025:15:37:06.409] 0_SNI_frontend SSL_backend/SSL_SERVER 1/0/167 3288 -- 6/4/3/2/0 0/0
2025-03-13T15:37:06-06:00	Informational	haproxy	123.123.123.123:35328 [13/Mar/2025:15:37:06.409] 1_HTTPS_Frontend/127.4.4.3:443: SSL handshake failure

It appears to try the HTTPS front end first, fail then tries the SNI. From what I understand the SNI should then be routing the traffic according to the rule to not SSL offload but it doesn't... 

Here is my config (sanitized of course/hopefully)

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    ocsp-update.mindelay 300
    ocsp-update.maxdelay 3600
    httpclient.resolvers.prefer   ipv4
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua
cache opnsense-haproxy-cache
    total-max-size 4
    max-age 60
    process-vary off

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 1_http_frontend ()
frontend 1_http_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive

    # logging options
    # ACL: NoSSL_condition
    acl acl_60ece619a266e9.71758723 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_60ece619a266e9.71758723

# Frontend: 0_SNI_frontend ()
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option tcplog
    option socket-stats

    # ACTION: PUBLIC_nooffloaddomain_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/67d34435367b99.58937721.txt)]

# Frontend: 1_HTTPS_Frontend ()
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60ed00e1c92857.09613107.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/615ce4557a4dc4.14466569.txt)]

# Backend: Plex_backend ()
backend Plex_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Plex 192.168.1.42:32400 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_SERVER 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Ombi_backend ()
backend Ombi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Ombi 192.168.1.84:5055

# Backend: HomeAssist_backend ()
backend HomeAssist_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server ha 192.168.1.12:8123

# Backend: storage_backend ()
backend storage_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    option forwarded
    option forwardfor
    server storage 192.168.1.69:443 ssl alpn h2,http/1.1 verify none

# Backend: nooffloaddomain_backend (nooffloaddomain)
backend nooffloaddomain_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server nooffloaddomain 192.168.1.118 ssl verify none resolve-prefer ipv4



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED

Code Select Expand

#615ce4557a4dc4.14466569
# public access subdomains
plex Plex_backend
storage storage_backend
ha HomeAssist_backend
workingdomain.com Ombi_backend

Code Select Expand

#67d34435367b99.58937721
# public access subdomains
notworkingdomain.com notworkingdomain_backend
staticstuff notworkingdomain_backend
I have no doubt I've missed something completely, or at the very least misunderstood and would appreciate any help that can be provided. 

Sorry I don't mean to hijack this thread, but my question is along these same lines. 
I too am running HA/CARP and for DHCP I'm using ISC. 
I have the failover IP set to the other opnsense IP as needed. 
In HA I have DHCPD included in my  XMLRPC Sync. 

This works out great and seems to keep the failover IP set correctly on both nodes. But is changing the failover split each time the sync runs (I have a cron job). 
The info on that field says on the backup I should leave it blank, but then each sync it changes this to match the primary (256). 
Question comes down to should I just remove DHCPd from the  XMLRPC Sync? Or is something else wrong....
Additional question should I also stop other services that have their own HA settings from being included in the  XMLRPC Sync? HAProxy as an example, and prob others I'm not thinking of?
Is there any kind of definitive list of applications that sync themselves vs using XMLRPC?

I'm having the same issue, or so it seems. 
if I start the control agent then both it and the keadhcpserver services stop and will not start. 
disable the control agent and dhcp server works fine (all in kea). 

I'm in the process of setting up my backup opnsense (HA). So I need to figure this out. 

I set the bind IP to 127.0.0.1 and the port to 8070 (8070 is not in use). 

Thanks! I continue on my journey then. 

New to HA with opnsense so learning. Used AI to help with some, but obviously that leaves things out.
I have gone over the documentation, and apparently am still missing something.
On my primary OPNsense I see the attached screen. Does this indicate that the backup is not ACTUALLY in "backup mode" since all the services appear to be running?

In https://docs.opnsense.org/manual/how-tos/wireguard-client-mullvad.html#
Under Step 1 there is a paragraph:
"On Peers tab create a new Peer, give it a Name, set 0.0.0.0/0 in Allowed IPs and set the DNS to 193.138.218.74. This is the one MulladVPN provides for privacy."
However in Peers in Opnsense there is not place to set DNS address...

In https://docs.opnsense.org/manual/how-tos/wireguard-selective-routing.html#step-6-create-a-gateway
There is a Blue note. Is the first paragraph still accurate, or did the option "Disable Host Route" fix this essentially?

Anyone solve this? I just started having the issue too.

I adjusted things quite a bit. I'm now consistently able to load plex.mydomain.com
but going to mydomain.com gives a 503 error

I got to this point using the amazing directions at:
https://forum.opnsense.org/index.php?topic=23339.0

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    maxconn                     1000
    tune.ssl.default-dh-param   4096
    spread-checks               0
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 80_redirect (HTTP Redirect)
frontend 80_redirect
    bind 0.0.0.0:80 name 0.0.0.0:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: HTTP
    acl acl_603205c7df9a30.32769784 req.ssl_ver gt 0

    # ACTION: http-redirect
    http-request redirect scheme https code 301 if !acl_603205c7df9a30.32769784

# Frontend: SNI_frontend ()
frontend SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: HTTPS_frontend ()
frontend HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/60e263758b8c80.28077203.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: Ombi
    acl acl_6029bf280a2d50.77237914 hdr_beg(host) -i ombi
    # ACL: plex
    acl acl_6032c230647965.95880753 hdr_beg(host) -i plex

    # ACTION: OmbiRules
    use_backend OmbiPool if acl_6029bf280a2d50.77237914
    # ACTION: PlexRules
    use_backend PlexPool if acl_6032c230647965.95880753

# Backend: OmbiPool ()
backend OmbiPool
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Ombi 192.168.1.42:3000 ssl verify none

# Backend: PlexPool ()
backend PlexPool
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex 192.168.1.42:32400 ssl verify none

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_SERVER 127.0.0.1 send-proxy-v2 check-send-proxy

listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# statistics are DISABLED





I've been finding the UI for haproxy in OPNSense more difficult to configure than it was in pfsense.
This tells me I really don't understand haproxy well enough, so if my question is something that should be understood I do apologize.
I have a domain mydomain.com
At this time I have 2 servers I'm trying to send through haproxy(plex and ombi).
I believe I have all the firewall rules doing what they need to for forwarding.

I'd like any traffic for plex.mydomain.com to always go to the plex server
Anything else, including someone just typing https://mydomain.com go to ombi
With the config I'm including I seem to get ombi working one restart, then plex working on the next restart. Almost like the rules are in some sort of priority order that switches each time I restart haproxy.
Once I have these 2 working I will eventually add other services.

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin expose-fd listeners
    nbproc                      1
    nbthread                    20
    maxconn                     10
    tune.ssl.default-dh-param   1024
    spread-checks               0
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0 info
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tls-tickets
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256

defaults
    log     global
    option redispatch -1
    maxconn 100
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: ombi (HTTPS Ombi)
frontend ombi
    bind 0.0.0.0:443 name 0.0.0.0:443 ssl no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256 crt-list /tmp/haproxy/ssl/6029c02e5cc836.63584345.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: Ombi
    acl acl_6029bf280a2d50.77237914 hdr(host) -i plex.mydomain.com

    # ACTION: OmbiRules
    use_backend OmbiPool if !acl_6029bf280a2d50.77237914

# Frontend: 80_redirect (HTTP Redirect)
frontend 80_redirect
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    option httplog
    # ACL: HTTP
    acl acl_603205c7df9a30.32769784 req.proto_http

    # ACTION: http-redirect
    http-request redirect scheme https if acl_603205c7df9a30.32769784

# Frontend: Plex (HTTPS Plex)
frontend Plex
    bind 0.0.0.0:443 name 0.0.0.0:443 ssl ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256 crt-list /tmp/haproxy/ssl/60de5b0e9afdd3.95063140.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: plex
    acl acl_6032c230647965.95880753 hdr(host) -i plex.mydomain.com

    # ACTION: PlexRules
    use_backend PlexPool if acl_6032c230647965.95880753

# Backend: OmbiPool ()
backend OmbiPool
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Ombi 192.168.1.42:3000

# Backend: PlexPool ()
backend PlexPool
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Plex 192.168.1.42:32400

listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# statistics are DISABLED

Disable routes is now allowing me to keep everything working regardless of the outbound rule. BUT now I can't get anything routed through wireguard.

I've tried following https://imgur.com/gallery/JBf2RF6. But when I did it just didn't work at all it seemed. I guess I don't understand the purpose in that link of using an alias to represent the "normal" network. And then at the very end it appears anything that is NOT part of the normal network is supposed to use regular WAN - very last rule under the words Optional Step...

I did find that thread and have been trying to follow.
It's anytime I get to the last 2 steps that i"m getting hung up
Create a NAT rule on the Mullvad interface for your LAN network - This was easy enough, but makes it so ALL traffic goes out WG0


Create a firewall rule for your LAN interface directing (selected) traffic to the Mullvad gateway (or the group in my case)



I'm not seeing why I need a gateway for WG0. Or how to route to it though.

I have OPNsense running as my firewall, all traffic on my network runs through it.
I've setup wireguard in OPNsense. It's just a client though, I use Torguard as the service providor for VPN. I am NOT trying to connect back into my network via wireguard. Just use wireguard to "protect" me.
Created the NAT-Outbound rule and all traffic on my LAN flows out via wireguard - this is good of course.
However I want to change it so only 1 IP on my network goes through the wireguard and everything else just goes out the WAN.
I'm using Hybrid in the outbound section.
Tried setting the source from LAN Net to the IP I want, but then all devices lose access to internet.
Also tried an alias(my ultimate end goal will be routing just an alias out my wireguard).

I've done searching but it's a mix of people wanting to connect to their own networks via wireguard and some only wanting entire network routed. If someone could point me to a how-to that's my situation, or similar, I'd really appreciate it!