Messages

@TheHellSite, many thanks and kudos for your tremendous effort, contribution and help!

I try to avoid seeking for help and solve my problems on my own. But after upgrading to 24.1 I stuck in the CRON configuration when it comes to the update HAProxy OCSP Data you mentioned in Part5.4, this feature has disappeared and can no longer be selected. I assume this is needed to get the OCSP must staple extension running.

Is there an alternative way of configuring or what I'm doing wrong or missing.

Thanks loop0

As i know the OCSP update cronjob isn't needed anymore since the OCSP feature was completely revamped with the actual version of haproxy 4.2 which is bundled in opnsense 24.1

I had some errors with the OCSP updates so i opened a issue in the opnsense/plugins github repo.
https://github.com/opnsense/plugins/issues/3755

First of all, a huge thank you to TheHellSite for this detailed tutorial!

Unfortunately, I need your help. I have configured HAProxy as described in the tutorial. However, with my own domain.

All services that are to be reached externally work as desired. Only the internal service does not seem to be "noticed" by HAProxy. Unfortunately, no accesses to the "node2-ipmi" service from the source IP from the "10.10.10.0/24" network appear in the log. I cannot connect to the service "node2-ipmi".

In firefox i got this warning "SEC_ERROR_UNKNOWN_ISSUER".

Since no log entries appear in the log, I cannot attach any.

Config export:

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (listening to 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_65612d875c4e55.24914702 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_65612d875c4e55.24914702

# Frontend: 1_HTTPS_frontend (listening to 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6561dfa723cb35.23136075.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    # ACL: LOCAL_SUBDOMAINS_FQDN_condition
    acl acl_6563927a593ba4.09519486 src domain.tld
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_65627ea0efa5d5.95729048 src 10.10.5.0/28 10.10.10.0/24 10.10.11.0/24
    # ACL: nextcloud_caldav
    acl acl_65626936202592.20944712 path_beg -i /.well-known/caldav
    # ACL: nextcloud_carddav
    acl acl_656269439b5220.54434789 path_beg -i /.well-known/carddav

    # ACTION: LOCAL_SUBDOMAINS_rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/656277f5815fc5.43737480.txt)] if acl_6563927a593ba4.09519486 || acl_65627ea0efa5d5.95729048
    # ACTION: PUBLIC_SUBDOMAINS_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/65612e0d931f69.06203948.txt)]
    # ACTION: nextcloud_dav
    http-request set-path /remote.php/dav if acl_65626936202592.20944712 || acl_656269439b5220.54434789

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: cloud_backend ()
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cloud_server 10.10.20.5:80

# Backend: vw_backend ()
backend vw_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server vw_server 10.10.20.7:80

# Backend: office_backend ()
backend office_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server office_server 10.10.20.8:80

# Backend: rezepte_backend ()
backend rezepte_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server rezepte_server 10.10.20.9:3000

# Backend: cash_backend ()
backend cash_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server cash_server 10.10.20.10:5006

# Backend: node2-ipmi_backend ()
backend node2-ipmi_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server node2-ipmi_server 10.10.5.6:443 ssl verify none



# statistics are DISABLED


With best regards,
techsolo12

First of all: I use Opnsense with Unbound and AdGuard.

Like some other users, I had the problem that when I configured the public service "0_SNI_fronted", the HAProxy service could no longer be started. If I only set port "443" the problem did not occur.
So the problem was that port "80" was already blocked by another service.

A look at "Interfaces -> Diagnostics -> Netstat -> Socket" revealed that port "80" was already being used by AdGuard.

To change this, I adjusted the default http port in the AdGuard config. From "80" to "81". To do this, simply stop the AdGuard service in the WebUI, then go to the shell and edit the config.

Code Select Expand

nano /usr/local/AdGuardHome/AdGuardHome.yaml

Now you can edit the default port to "81".

Code Select Expand

http:
  pprof:
    port: 6060
    enabled: false
  address: 0.0.0.0:81
  session_ttl: 720h

After all you start the AdGuard service on WebUI. Now you should can configure services in haproxy.

best regards,
techsolo12

Hello everyone,

I hope I can find help for my problem here. I have been working on setting up a VPN tunnel from a VPS to my Homelab for a few days.
The tunnel itself is not the problem. The tunnel is up and running, I can also reach all participants via ping and a "nmap" on port 80 and 443 from the VPS in the direction of the reverse proxy is also successful.
In my opinion, there is no response from the reverse proxy.

If any of you can find the time to look at the problem, I would be very grateful!
If you have any questions or need information, please let me know.

First a few details.
The VPS serves as a wireguard server (10.10.90.1). The Opnsense (10.10.90.2) connects to the VPS as a client.
Behind the Opnsense is a reverse proxy (10.10.20.4). All incoming requests on port 80 and 443 on the VPS should be forwarded to the reverse proxy.

Enclosed is the Wireguard config:

Code Select Expand

[Interface]
PrivateKey = ----
ListenPort = 1195
Address = 10.10.90.1/28

PostUp = iptables -A FORWARD -i ens6 -o wg0 -p tcp --syn -m multiport --dports 80,443 -m conntrack --ctstate NEW -j ACCEPT
PostUp = iptables -A FORWARD -i wg0 -o ens6 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostUp = iptables -t nat -A PREROUTING -i ens6 -p tcp -m multiport --dports 80,443 -j DNAT --to-destination 10.10.20.4
PostUp = iptables -t nat -A POSTROUTING -o wg0 -p tcp -m multiport --dports 80,443 -d 10.10.20.4 -j SNAT --to-source 10.10.90.1

PostDown = iptables -D FORWARD -i ens6 -o wg0 -p tcp --syn -m multiport --dports 80,443 -m conntrack --ctstate NEW -j ACCEPT
PostDown = iptables -D FORWARD -i wg0 -o ens6 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
PostDown = iptables -t nat -D PREROUTING -i ens6 -p tcp -m multiport --dports 80,443 -j DNAT --to-destination 10.10.20.4
PostDown = iptables -t nat -D POSTROUTING -o wg0 -p tcp -m multiport --dports 80,443 -d 10.10.20.4 -j SNAT --to-source 10.10.90.1

[Peer]
PublicKey = ----
PresharedKey = ----
AllowedIPs = 10.10.90.2/32, 10.10.20.4/32
PersistentKeepalive = 25


The following settings have been made on the Opnsense:
VPN -> Wireguard -> Instances:

Code Select Expand

Tunnel addresse = 10.10.90.2/28
Peers = proxy-vpn
Disable Routes = yes
Gateway = 10.10.90.1

VPN -> Wireguard -> Peers:

Code Select Expand

Name = proxy-vpn
Allowed IPs = 10.10.90.1/32
Endpoint Address = IP address of the VPS
Endpoint Port = 1195

Firewall -> Rules -> Wireguard_proxyvpn:

Code Select Expand

IPv4 TCP, S: 10.10.90.1, P: *, D: 10.10.20.4, P: 80
IPv4 TCP, S: 10.10.90.1, P: *, D: 10.10.20.4, P: 443
IPv4 ICMP, S: 10.10.90.1, P: *, D: 10.10.20.4, P: *
IPv4 TCP/UDP, S: Wireguard_proxyvpn net, P: *, D: Wireguard_proxyvpn net, P: 80

Firewall -> Rules -> DMZ (Reverse Proxy Network)

Code Select Expand

IPv4 ICMP, S: 10.10.20.4, P: *, D: Wireguard_proxyvpn net, P: *

System -> Gateways -> Single:

Code Select Expand

N: Wireguard_proxyvpn, I: Wireguard_proxyvpn, G:10.10.90.1, MIP: 10.10.90.1

This option is also activated on the VPS.

Code Select Expand

net.ipv4.ip_forward=1

Here the same. i cannot can start haproxy, after a click on start the pages refreshed and no daemon is started. Also not log entry or something else.

[Servername]

Hello Guys,

at first, a big thank you to @yeraycito for your tutorial!

I have some addional steps for the tutorial, if you want to use a wildcard certificate from acme client and domain name in your local network.
For me i can't get adguard webui with ssl working on the domain name from opnsense. My goal was to use the webui like this: https://opnsense.your-local-domain.tld or on a another port like opnsense.your-local-domain.tld:4443 with ssl wildcard certificate.


Opnsense 22.7.4 Install:

1 - Activate mimugmail's community repository

2 - Install AdGuardHome from System --> Firmware --> Plugins

3 - Opnsense - System - Settings -General

      DNS Servers: empty

      Untick: Do not use the local DNS service as a nameserver for this system

      Untick: Allow DNS server list to be overridden by DHCP/PPP on WAN

4 - Services – DHCPv4 – [LAN] : DNS Servers all empty

5 – Opnsense – Services - Unbound DNS – General

       Tick: Enable Unbound ( Listen Port: 5353 )

       Tick: Enable DNSSEC Support
       
       Network Interfaces: All

6 - Opnsense - Services - Unbound - Dns Over Tls

      Server IP: 1.1.1.1

      Server Port: 853

      Verify CN: cloudflare-dns.com

7 - Open SSH Tunnel to OPNSense and edit the following config file
     

Code Select Expand

nano /usr/local/AdGuardHome/AdGuardHome.yaml

7.1 - You need to change following parts:

Code Select Expand

bind_host: 0.0.0.0
dns:
  bind_hosts:
    - 0.0.0.0


7.2 - Activate and start AdGuardHome from Services --> AdGuardHome

8 - Navigate to http://Opnsense ip:3000/ ( 127.0.0.1:3000 ) to complete the setup Adguard

9 - Adguard Home - DNS Configuration - Upstream Servers:

       Add 127.0.0.1:5353    !!!Delete those that exist!!!

10 – Adguard Home – DNS Configuration – Bootstrap DNS servers

       Add 127.0.0.1:5353    !!!Delete those that exist!!!
     
11 - Adguard Home - DNS Configuration - Private reverse DNS servers:

       Add 127.0.0.1:5353

12 - Now go Settings -> Encryption

       Pick "Encrytion activation"

       Servername = opnsensehostname.your-local-domain.tld

       Tick: "Automatic HTTPS redirect"
 
       If you want change HTTPS do it in "HTTPS-Port"

       Under Certificate choose your certificate from acme client which should be located in:

Code Select Expand

/var/etc/acme-client/home/*.your-local-domain.tld/fullchain.cer

       Under Private Key choose your certificate from acme client which should be located in:

Code Select Expand

/var/etc/acme-client/home/*.your-local-domain.tld/*.your-local-domain.tld.key

13 - Save the settings
       
If you get a warninghint like: "validating certificate pair: certificates has no IP addresses; DNS-over-TLS won't be advertised via DDR" it's a known bug since version 0.127.16 with update to version 0.127.19 the color is only white and not red like in x.16, x.17, x.18

With best regards
techsolo12

EDIT: If you want to check if your setup works correctly you can use this website https://www.cloudflare.com/de-de/ssl/encrypted-sni/

Hello Guys!

Today its my first post here at this forum. At first @TheHellSite THANK YOU for your tutorial it helps my a lot! Before i used nginx proxy manager which was a lot easier than haproxy :)

I had one for my big problem and need the help from you all, please. I want to configure vaultwarden with websocket support in haproxy. The normal redirect to vaultwarden is no problem, but to add websocket support is still driven my crazy!

Sorry, but out of scope of this tutorial. Please ask in the official HAProxy forum.

Hello Guys!

Unfortunally nobody in the other forums can help me with this situation. Anybody in vaultwarden or haproxy forum. Is here nobody who had vaultwarden getting worked? :(

With best regards;
techsolo12

Hello Guys!

Today its my first post here at this forum. At first @TheHellSite THANK YOU for your tutorial it helps my a lot! Before i used nginx proxy manager which was a lot easier than haproxy :)

I had one for my big problem and need the help from you all, please. I want to configure vaultwarden with websocket support in haproxy. The normal redirect to vaultwarden is no problem, but to add websocket support is still driven my crazy!

https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples
Here are some examples how the proxy setup should, but i dont understand were my problem is.

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.0.0.1:80)
frontend 1_HTTP_frontend
    bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_636976fd9d4d71.97561865 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_636976fd9d4d71.97561865

# Frontend: 1_HTTPS_frontend (Listening on 127.0.0.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/636aad8d3cbe18.58884679.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: nc_carddav
    acl acl_636ba4e5b6aa82.28881573 path_end -i /.well-known/carddav
    # ACL: nc_caldav
    acl acl_636ba2d9f14933.27250118 path_end -i /.well-known/caldav
    # ACL: vw_ws_acl01_condition
    acl acl_636c2f2b5accd9.55827620 path_beg -i /notifications/hub
    # ACL: vw_ws_acl02_condition
    acl acl_636cc909734817.72974823 path_beg -i /notifications/hub/negotiate
    # ACL: vw_ws_acl03_condition
    acl acl_636ccac64fcd74.27409543 path_beg -i /notifications/hub
    # ACL: vw_ws_acl04_condition
    acl acl_636ccae443ca48.73072029 path_beg -i /notifications/hub/negotiate

    # ACTION: nc_carddav_rule
    http-request redirect code 301 location /remote.php/dav if acl_636ba4e5b6aa82.28881573
    # ACTION: nc_caldav_rule
    http-request redirect code 301 location /remote.php/dav if acl_636ba2d9f14933.27250118
    # ACTION: PUBLIC_SUBDOMAINS-map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63653d33935cd3.47503593.txt)]
    # ACTION: vw_ws_acl01_rule
    use_backend vw_backend if !acl_636c2f2b5accd9.55827620
    # ACTION: vw_ws_acl02_rule
    use_backend vw_backend if acl_636cc909734817.72974823
    # ACTION: vw_ws_acl03_rule
    use_backend vw_ws_backend if acl_636ccac64fcd74.27409543
    # ACTION: vw_ws_acl04_rule
    use_backend vw_ws_backend if !acl_636ccae443ca48.73072029

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.0.0.1 send-proxy-v2 check-send-proxy

# Backend: office_backend (Onlyoffice)
backend office_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server office_server 10.10.20.8:80

# Backend: vw_backend (Vaultwarden)
backend vw_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server vw_server 10.10.20.7:80

# Backend: mc_backend (Minecraft Server)
backend mc_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server mc_server 10.10.40.4:80

# Backend: cloud_backend (Nextcloud01)
backend cloud_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server cloud_server 10.10.20.5:80

# Backend: demo_backend (Nextcloud02)
backend demo_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server demo_server 10.10.20.6:80

# Backend: kunden_backend (Nextcloud03)
backend kunden_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server kunden_server 10.10.20.11:80

# Backend: vw_ws_backend (Vaultwarden Websocket)
backend vw_ws_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server vw_ws_server 10.10.20.7:3012



# statistics are DISABLED

Code Select Expand

2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (1_HTTPS_frontend/HTTP)
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (1_HTTPS_frontend/HTTP)
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (1_HTTPS_frontend/HTTP)
2022-11-10T13:33:48 Informational haproxy Connect from 10.10.10.239:54010 to PUBLICIP:443 (0_SNI_frontend/TCP)

With best regard,
techsolo12