Messages

[EDIT]

EDIT

Nevermind, my first issue was related to https://forum.opnsense.org/index.php?topic=38435.0 which has since been patched.

As always, i appreciate the upkeep of this guide.

hi guys,

I am trying to setup a SSO using keycloak, openLDAP and other providers. I have it all working locally but now wanting to communicate to the LDAP server via ldap://ldap.mydomain.com instead of ldap://192.168.1.104:1389.

The only difference that i have tried is changing the mode on the backend server from HTTP to TCP. I just recieve a timeout error when testing.

Code Select Expand

2022-09-24 13:40:25,284 ERROR [org.keycloak.services] (executor-thread-39) KC-SERVICES0055: Error when connecting to LDAP: ldap.mydomain.com:389: javax.naming.CommunicationException: ldap.mydomain.com:389 [Root exception is java.net.SocketTimeoutException: connect timed out]

Code Select Expand


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     100000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62bbec3b1189e7.31090598 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62bbec3b1189e7.31090598

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62bbef8e4ab6b5.77631912.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62bbecc24b7a71.66647551.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: PRISM_backend ()
backend PRISM_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server PRISM_server 192.168.1.103:2342

# Backend: REQUEST_backend ()
backend REQUEST_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server REQUEST_server 192.168.1.104:5055

# Backend: LDAP_backend ()
backend LDAP_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server LDAP_server 192.168.1.104:1636 ssl verify none


is there a way to check this via the shell window or re enable it?

I cant seem to be able to access the web gui @ http://192.168.1.1/ or https://192.168.1.1/ page returns

This page isn't working192.168.1.1 didn't send any data.
ERR_EMPTY_RESPONSE

I am 100% on the same submask. I restarted opnsense, connected a monitor and logged in. The interfaces are correctly assigned and i have internet wan access. I was unsure if it was something i changed the last time i connected. So I've I restored a backup but still the same.

i just cant access the web gui login

Although it has the same outcome, The steps you provided using only 1 map file is a lot cleaner and easier to follow.

Thanks again for pointing this out.

I am not sure if this is the correct way to achieve multiple domains pointing to different backends but it seems to be working for me.
At first I ran into a issue were all domains could access the same subdomain, this is when I realized I just needed some extra conditions.

Here are the steps to achieve; service.example.com & service1.example1.com

Services --> ACME Client --> Certificates
Add the certificate for your extra domains and forcefully issue your certificate

Services --> HAProxy --> Settings --> Advanced --> Map Files
Here we will create a new map file for each domain "PUBLIC_SUBDOMAINS_map-example" & "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Rules & Checks --> Conditions
Add a new condition for each domain that you have added.
Name = "example1_condition"
Description = "Traffic matches example1.com"
Condition type = "host contains"
Host Contains = "example1.com"

Services --> HAProxy --> Settings --> Rules & Checks --> Rules
Add a map rule for each domain while also selecting our newly created "example1_condition"
Name = "PUBLIC_SUBDOMAINS_map-rule-example1"
Select conditions = "example1_condition"
Map file = "PUBLIC_SUBDOMAINS_map-example1"

Services --> HAProxy --> Settings --> Virtual Services --> Public Services
Finally we edit our "1_HTTPS_frontend"
Add all extra domains in the "Certificates" input.
Scroll down and add each map-rule-example1 in the "Select Rules" input

I was hoping someone can help me get the live server extension working within the code-server docker. I rely on being able to see my changes live and am struggling to access the hosted port 5500.cs.example.com

I have HAProxy working with ACME & Cloudflare. Following this great tutorial (https://forum.opnsense.org/index.php?topic=23339.0). I can access code-server via cs.example.com but when I launch the live server extension, I should be able to access this via 5500.cs.example.com.

by adding a extra port in the container I am able to locally access live server via, http://serverip:5500. I tried to create a new backend/front end pointing here but this did not work.

Working setup sofar
Cloudflare = A record, *, IP
ACME > Certificates = Common name, *.example.com
HAProxy > front end & backend points to cs local ip & port

Does HAProxy have something similar to the EXTRA_DOMAINS argument that is referenced below?

After searching the LSIO discord I found the below support using their SWAG container but I cant implement this with my setup (HAProxy).

aptalca — 16/01/2022
you just add *.code-server.domain.com into EXTRA_DOMAINS in SWAG arguments, create a wildcard cname for *.code-server and set the server address in code server env vars PROXY_DOMAIN. After that code server will let you access any port at https://port.code-server.domain.com

MuadDDib — 16/01/2022
I'm using CloudFlare DNS, so I have my A domain for domain.com then a CNAME for code.domain.com

aptalca — 16/01/2022
you need another cname for *.code.domain.com

MuadDDib — 16/01/2022
Now I have to create a new CNAME for *.code.domain.com like the image I'll show next

aptalca — 16/01/2022
if you're using code instead of code-server, you'll have to edit the proxy conf for code server to edit both instances of code-server to code on this line:

aptalca — 16/01/2022
what validation method are you using? http?

MuadDDib — 16/01/2022
yep
I thought that was the de facto for CF DNS

aptalca — 16/01/2022
ah, then you can't do wildcard. change the cname to 5500.code

MuadDDib — 16/01/2022
ok give me a sec

aptalca — 16/01/2022
and add 5500.code.domain.com to EXTRA_DOMAINS
only port 5500 will work, but it will get you go live
I gotta go to bed

MuadDDib — 16/01/2022
@aptalca just wanted to let you know that it worked almost flawlessly!

Try to use "Source IP matches a specific IP" instead

I actually set this first time around which gave the same output. So I then decided to try a broader range with "IP is local".

I tried again "Source IP matches a specific IP" of the specific subnet again with no luck. have even tried isolating it to the IP of the client I am currently using.

the counter still is not being passed on the final stage of the route. After checking the HAProxy log file, as I navigate to local.website.com i am receiving a external request from cloudflare. I suspect the issue lies within the browser redirecting the request out or HAproxy not grabbing it locally before it goes out??

I will make my way over to the other thread you linked then as it is a similar issue.

I am on 22.1.5 and after checking the general log I also have the below

Code Select Expand


2022-04-14T16:42:58 Error opnsense /firewall_virtual_ip.php: The command `/sbin/ifconfig 'lo0' inet '192.168.64.1' -alias' failed to execute



--EDIT--
After reading the linked thread, a patch was applied in 22.1.4 but you also need to untick "Allow service binding".
I can confirm this has fixed the issue. Thanks again for point me in the right direction.

---EDIT 2---
I am now interested in the last part of the tutorial, I did not have anything like this setup on my previous pfsense setup so it will be a great addition. It would make for remembering local ip/ ports easier when navigating to locally hosted services.
I have followed the steps to make these subdomains accessible only from my local network but getting a "503 service unavailable".

I think its not reading the local map file correctly. I followed the previious steps for tracing and can see the counters go up when I try to access local.website.com but nothing is being passed to the prism_backend.

Code Select Expand


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: LOCAL_SUBDOMAINS_SUBNETS_condition
    acl acl_6257dfacde7e16.43417850 src_is_local

    # ACTION: LOCAL_SUBDOMAINS_map-rule
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6257d684d34507.32920094.txt)] if acl_6257dfacde7e16.43417850
    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none

# Backend: Prism_backend ()
backend Prism_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server Prism_server 192.168.1.103:2342


PUBLIC_SUBDOMAINS_map

Code Select Expand


# public access subdomains
mineos MineOS_backend

LOCAL_SUBDOMAINS_map

Code Select Expand


# local access subdomains
prism Prism_backend

# public access subdomains
mineos MineOS_backend


just to confirm;
mineos.website.com > works locally and externally
prism.website.com > 503 error locally and externally

You guys are awesome, really appreciate you explaining the process. It is easy to follow along to a guide but to understand what is happening makes it that much easier down the line.

After pinging the VIP 192.168.64.1, it was timing out.

Checked the setting and all is correct to the tutorial.
Decided to changed the submask from 32 to 24.
Then I was able to ping the VIP and access my web services.
Reset the submask back to 32 and i am still able to ping the VIP and web services working with HTTP & HTTPS listening on the VIP. Very strange but it seems resolved.

Okay i can now access my webservices but in doing so a missed out the Virtual IP step.
After thinking about my issue some more I am listening HTTP & HTTPS traffic on 192.168.64.1 which I think is when the time out happens.

So I;
Service > HAProxy > Settings > Real Servers > SSL_server: changed FQDN or IP, from 192.168.64.1 to 192.168.1.1

Service > HAProxy > Settings > Virtual Services > 1_HTTPS_frontend: changed Listen Addresses, from 192.168.64.1:443 to 192.168.1.1:443

Service > HAProxy > Settings > Virtual Services > 1_HTTP_frontend: changed Listen Addresses, from 192.168.64.1:80 to 192.168.1.1:80

Now it is all working, What did I do wrong in setting up the Virtual IP I wonder.

0_SNI_frontend > Listen Addresses:0.0.0.0:80, 0.0.0.0:443
should this need to be the Virtual IP as opnsense runs on 192.168.1.1

^^fyi thankyou for the tips on tracing

This week i have moved away from pfSense, I had acme, cloudflare & HAProxy working prior to the switch. Installed opnsense while slowly getting my services back online I came across this well written tutorial which seems more in-depth than my old setup but run into issues while accessing the hosted web service, it is failing to load with a 522 error, the connection if timing out before a response I think?

I have not got any further in the guide than part 5, step 10. Accessing from outside of my network as this is not possible so far.

I have a static WAN IP.. in cloudflare a have [A record *.example.com > Static IP]

I have double checked all the settings in this tutorial and after some googling i came across a reddit post, suggesting they fixed the 522 error in opnsense because HAProxy wasn't listening on port 80 during the HTTPtoHTTPS redirect. Is there a way I can diagnose this issue and trace the route somehow.

Lastly

ACME do not show any error in the log files. 

Code Select Expand


2022-04-13T18:53:42 php AcmeClient: running automation (configd): Restart HAProxy
2022-04-13T18:53:42 php AcmeClient: running automations for certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: updated ACME X.509 certificate: *.example.com
2022-04-13T18:53:42 opnsense AcmeClient: successfully issued/renewed certificate: *.example.com
2022-04-13T18:51:27 opnsense AcmeClient: using challenge type: CloudFlare_DNS-01
2022-04-13T18:51:27 opnsense AcmeClient: account is registered: example.com
2022-04-13T18:51:27 opnsense AcmeClient: using CA: letsencrypt
2022-04-13T18:51:27 opnsense AcmeClient: issue certificate: *.example.com


HAProxy has no errors in the log file either

Code Select Expand


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    2
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: 1_HTTP_frontend (Listening on 192.168.64.1:80)
frontend 1_HTTP_frontend
    bind 192.168.64.1:80 name 192.168.64.1:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: NoSSL_condition
    acl acl_62565b172acae6.05588153 req.ssl_ver gt 0

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_62565b172acae6.05588153

# Frontend: 1_HTTPS_frontend (Listening on 192.168.64.1:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 192.168.64.1:443 name 192.168.64.1:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/62565eb5d0ff12.02152772.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/62565c00b116b3.27816426.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    server SSL_server 192.168.64.1 send-proxy-v2 check-send-proxy

# Backend: MineOS_backend ()
backend MineOS_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server MineOS_server 192.168.1.103:8443 ssl verify none