Messages

The bottom line for missing traffic is that it's not seen by firewall, tcpdump or netmap.

I'd say the main problem with Tailscale on FreeBSD is here: https://github.com/tailscale/tailscale/issues/5573

There are some weird hacks with exporting TS_DEBUG_NETSTACK_SUBNETS=0 suggested, did not look into it. As it as, I'd say assigning the interfaces is absolutely pointless, these cannot be treated the usual way, firewalled via pf nor can Tailscale be used for site-to-site VPNs on BSD.

Thx, that seems to be the issue. The work around looks difficult and hacky. Maybe i go back  to wireguard again..

I was wondering if your setup with a tailscale interface is working as you intended? I have a tailscale setup working but firewall rules are not working. I see no bytes flowing through the tailscale interface in traffic reporting. What i would like to do to is protecting the tailscale interface with zenarmor.

I had the same notifications the last couple of days (after upgrading zenarmor?) I have run zenarmor for almost a year and never had this notification. Don't know either how to troubleshoot

Thx to the excellent tutorial of @TheHellSite (https://forum.opnsense.org/index.php?topic=23339.225) I have HAproxy working.

I have one service I would like to be entered only through Authelia, to enable 2FA. I see a guide for pfsense (https://dkict.com/pfsense-haproxy-authelia/) but I cannot get it to work in OPNsense

I have the needed lua scripts in place but I cannot find a place to enter the needed configuration for the backend service like "acl remote_user_exist var(req.auth_response_header.remote_user) -m found"

Anyone has this kind of setup working and would like to help me

Strange but 503 error appear to me as well.
I tested with apache,nodejs,wamp nothing worked. They i try to redirect to my switch to see if my windows is not the problem... but nope.
DynamicDNS is configured and working fine,
All gui redirections disabled and opnsense gui port changed.
Added firewall rule to WAN , and no additional LAN rules added ( it's almost fresh install )
Acme - generated fine cert via dns. ( 2/4/2023, 7:23:39 PM   OK   2/4/2023, 7:23:40 PM )

Tested from external network via smartphone on cellular data.

One thing is that i am using proxmox to virtualize opnsense as "routerOnStick/Forbidden Router" and i pass two ports from quad NIC on promox-server as LAN/WAN for opnsense , and lan is going to dumb switch that transfer vlans/lan to rest of my house , so far not a single problem with that but maybe just maybe..

Code Select Expand

OPNsense 23.1_6-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022

Code Select Expand

HAProxy version 2.6.7-c55bfdb 2022/12/02 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.7.html
Running on: FreeBSD 13.1-RELEASE-p5 FreeBSD 13.1-RELEASE-p5 stable/23.1-n250372-c4ad069e50a SMP amd64

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:443 and 0.0.0.0:80)
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_backend

    # logging options
    option log-separate-errors
    option tcplog

# Frontend: 1_HTTP_frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    option log-separate-errors
    option httplog
    # ACL: NoSSL_condition
    acl acl_63dea06740dee5.93056632 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_63dea06740dee5.93056632

# Frontend: 1_HTTPS_frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63dea303583a84.37941891.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options
    option log-separate-errors
    option httplog

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63dea0bbafdf17.31648976.txt)]

# Backend: SSL_backend ()
backend SSL_backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: XKP_backend ()
backend XKP_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server XKP_server 192.168.1.104:80 ssl verify none



# statistics are DISABLED

Code Select Expand

2023-02-05T10:45:38 Error haproxy ********:31073 [05/Feb/2023:10:45:38.668] 1_HTTPS_frontend~ 1_HTTPS_frontend/<NOSRV> -1/-1/-1/-1/0 503 217 - - SC-- 2/1/0/0/0 0/0 "GET https://[********:/favicon.ico HTTP/2.0"

Code Select Expand

root@firewall:~ # haproxy -f /usr/local/etc/haproxy.conf -d
Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.

Available filters :
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Using kqueue() as the polling mechanism.
00000000:0_SNI_frontend.accept(0004)=0014 from [********:31207] ALPN=<none>
00000001:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000001:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/ HTTP/2.0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: cache-control: max-age=0
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: upgrade-insecure-requests: 1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: cross-site
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: navigate
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-user: ?1
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: document
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000001:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000001:1_HTTPS_frontend.clicls[0017:ffff]
00000001:1_HTTPS_frontend.closed[0017:ffff]
00000002:1_HTTPS_frontend.accept(0007)=0017 from [********:31207] ALPN=h2
00000002:1_HTTPS_frontend.clireq[0017:ffffffff]: GET https://********/favicon.ico HTTP/2.0
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: host: ********
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: dnt: 1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-mobile: ?1
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: save-data: on
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: user-agent: Mozilla/5.0 (Linux; Android 13; Mi 9T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-ch-ua-platform: "Android"
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-site: same-origin
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-mode: no-cors
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: sec-fetch-dest: image
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: referer: https://********/
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-encoding: gzip, deflate, br
00000002:1_HTTPS_frontend.clihdr[0017:ffffffff]: accept-language: en-US,en;q=0.9
00000002:1_HTTPS_frontend.clicls[0017:ffff]
00000002:1_HTTPS_frontend.closed[0017:ffff]
00000000:SSL_backend.srvcls[0014:ffff]
00000000:SSL_backend.clicls[ffff:ffff]
00000000:SSL_backend.closed[ffff:ffff]


I see also in your backend for XKP SSL is checked. Could you try by unchecking the SSL?

I do not understand what you mean by serving SSL on the HTTP port. I think I followed your tutorial to the letter (except for using a Let's encrypt certificate by using cloudflare API from my domain)

It is dangerous to do things like exposing services to the internet when you don't even understand this simple question from me!  :-\

Read step 9 of my FAQ. You should also really read the explanation of the "SSL checkbox" in the server setup page!
I bet you are not accessing your services by their local ip using HTTPS you are likely accessing them using HTTP.

Yes that was the problem...

Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.

Please give further details on what is and whet it is not working.

Are you able to access your services via their domain name from a device outside of your local network?

Did you configure the DNS overrides for the local clients?

Also your Bitwarden server seems to be misconfigured are you sure it is serving SSL on the HTTP port? Also verify this for your other service.

Well, nothing is working, both not from within the local network and also not from outside.

I did configure the DNS override, but I first try to access the services from my mobile device.

I do not understand what you mean by serving SSL on the HTTP port. I think I followed your tutorial to the letter (except for using a Let's encrypt certificate by using cloudflare API from my domain)

Edit: I found it, I needed to uncheck the SSL tickbox in the real server settings. In your tutorial you have it checked and I saw in this forum someone else who had the same problem...

Thx alot!!

Thx you @TheHellSite for this tutorial. Unfortunately I cannot get it to work. I always get a "503 service unavaible status"

Here is my config

Code Select Expand


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   4096
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: 1_HTTP_Frontend (Listening on 127.4.4.3:80)
frontend 1_HTTP_Frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor

    # logging options
    # ACL: NoSSL_condition
    acl acl_63de5470175f22.54470191 ssl_fc

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_63de5470175f22.54470191

# Frontend: 0_SNI_Frontend (Listening on 0.0.0.0:80, 0.0.0.0:443)
frontend 0_SNI_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL_Backend

    # logging options

# Frontend: 1_HTTPS_Frontend (Listening on 127.4.4.3:443)
frontend 1_HTTPS_Frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/63de597c094f01.72503480.certlist
    mode http
    option http-keep-alive
    option forwardfor
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map-rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/63de5520a92049.75714996.txt)]

# Backend: SSL_Backend ()
backend SSL_Backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: BITWARDEN_backend ()
backend BITWARDEN_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server BITWARDEN 192.168.2.55:80 ssl verify none

# Backend: CALIBRE_backend ()
backend CALIBRE_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server CALIBRE 192.168.2.40:8083 ssl verify none



# statistics are DISABLED

If I use the internal addresses of the websites (192.168.2.40:8083 for example) I can login. But using the website.domain.com I get a 503 error. In the logs I see that the client is accessing the public Ip on port 443.