Messages

That is ssh. Its a screenshot from my terminal, not on the console.

OPNsense just completely died, wouldn't even shutdown after pressing the power button (which usually does a clean shutdown and power off).

Took the power off, hooked up a monitor and powered it back on. Its actually a little better. More responsive and ping replies are better.

Code Select Expand


64 bytes from 10.0.0.1: icmp_seq=1771 ttl=64 time=5.912 ms
64 bytes from 10.0.0.1: icmp_seq=1772 ttl=64 time=6.739 ms
64 bytes from 10.0.0.1: icmp_seq=1773 ttl=64 time=4.854 ms
64 bytes from 10.0.0.1: icmp_seq=1774 ttl=64 time=7.470 ms
64 bytes from 10.0.0.1: icmp_seq=1775 ttl=64 time=7.027 ms
64 bytes from 10.0.0.1: icmp_seq=1776 ttl=64 time=5.839 ms
64 bytes from 10.0.0.1: icmp_seq=1777 ttl=64 time=18.693 ms
64 bytes from 10.0.0.1: icmp_seq=1778 ttl=64 time=5.696 ms
64 bytes from 10.0.0.1: icmp_seq=1779 ttl=64 time=10.358 ms
64 bytes from 10.0.0.1: icmp_seq=1780 ttl=64 time=68.548 ms
64 bytes from 10.0.0.1: icmp_seq=1781 ttl=64 time=81.970 ms
64 bytes from 10.0.0.1: icmp_seq=1782 ttl=64 time=4.834 ms
64 bytes from 10.0.0.1: icmp_seq=1783 ttl=64 time=13.632 ms
64 bytes from 10.0.0.1: icmp_seq=1784 ttl=64 time=29.814 ms
64 bytes from 10.0.0.1: icmp_seq=1785 ttl=64 time=7.193 ms
64 bytes from 10.0.0.1: icmp_seq=1786 ttl=64 time=4.040 ms
64 bytes from 10.0.0.1: icmp_seq=1787 ttl=64 time=4.211 ms
64 bytes from 10.0.0.1: icmp_seq=1788 ttl=64 time=26.885 ms
64 bytes from 10.0.0.1: icmp_seq=1789 ttl=64 time=8.788 ms
64 bytes from 10.0.0.1: icmp_seq=1790 ttl=64 time=7.044 ms
64 bytes from 10.0.0.1: icmp_seq=1791 ttl=64 time=10.948 ms
64 bytes from 10.0.0.1: icmp_seq=1792 ttl=64 time=4.936 ms
64 bytes from 10.0.0.1: icmp_seq=1793 ttl=64 time=27.574 ms
64 bytes from 10.0.0.1: icmp_seq=1794 ttl=64 time=25.530 ms
64 bytes from 10.0.0.1: icmp_seq=1795 ttl=64 time=8.949 ms


Still not good, but better. Still some timeouts too, less than before though.

Attached. Sometimes I see this one:
37650 root          2  52    0    70M    49M usem     0   0:01  42.22% python3.9

But that's it. Less than 100kbps traffic in total on the graph.

Rebooted, no change.

Disabled haproxy and wireguard services too. Average less than 50kbps now. I don't understand what is going on  ::)

I waited for the fixes on HAproxy with SNI to update. Seems like there's solution so I decided to backup and upgrade. The upgrade went fine, but OPNsense is so incredibly slow its crazy.

I have a ping open and when I click to go to the Dashboard this happens:

Code Select Expand


64 bytes from 10.0.0.1: icmp_seq=548 ttl=64 time=331.020 ms
64 bytes from 10.0.0.1: icmp_seq=549 ttl=64 time=7.583 ms
64 bytes from 10.0.0.1: icmp_seq=550 ttl=64 time=54.139 ms
64 bytes from 10.0.0.1: icmp_seq=551 ttl=64 time=545.836 ms
64 bytes from 10.0.0.1: icmp_seq=552 ttl=64 time=459.255 ms
64 bytes from 10.0.0.1: icmp_seq=553 ttl=64 time=270.995 ms
64 bytes from 10.0.0.1: icmp_seq=554 ttl=64 time=61.969 ms
64 bytes from 10.0.0.1: icmp_seq=555 ttl=64 time=17.350 ms
64 bytes from 10.0.0.1: icmp_seq=556 ttl=64 time=35.384 ms
64 bytes from 10.0.0.1: icmp_seq=557 ttl=64 time=226.723 ms
64 bytes from 10.0.0.1: icmp_seq=558 ttl=64 time=70.730 ms
64 bytes from 10.0.0.1: icmp_seq=559 ttl=64 time=458.012 ms
64 bytes from 10.0.0.1: icmp_seq=560 ttl=64 time=25.976 ms
Request timeout for icmp_seq 561
64 bytes from 10.0.0.1: icmp_seq=562 ttl=64 time=614.986 ms
Request timeout for icmp_seq 563
Request timeout for icmp_seq 564
64 bytes from 10.0.0.1: icmp_seq=563 ttl=64 time=2459.566 ms
Request timeout for icmp_seq 566
Request timeout for icmp_seq 567
Request timeout for icmp_seq 568
Request timeout for icmp_seq 569
Request timeout for icmp_seq 570
Request timeout for icmp_seq 571
Request timeout for icmp_seq 572
Request timeout for icmp_seq 573
64 bytes from 10.0.0.1: icmp_seq=564 ttl=64 time=10620.278 ms
64 bytes from 10.0.0.1: icmp_seq=565 ttl=64 time=9665.505 ms
64 bytes from 10.0.0.1: icmp_seq=566 ttl=64 time=8736.085 ms
64 bytes from 10.0.0.1: icmp_seq=567 ttl=64 time=7975.989 ms
64 bytes from 10.0.0.1: icmp_seq=568 ttl=64 time=7223.718 ms
64 bytes from 10.0.0.1: icmp_seq=569 ttl=64 time=6320.197 ms
64 bytes from 10.0.0.1: icmp_seq=570 ttl=64 time=5447.343 ms
64 bytes from 10.0.0.1: icmp_seq=571 ttl=64 time=4457.011 ms
64 bytes from 10.0.0.1: icmp_seq=572 ttl=64 time=3555.876 ms
64 bytes from 10.0.0.1: icmp_seq=573 ttl=64 time=2556.972 ms
64 bytes from 10.0.0.1: icmp_seq=574 ttl=64 time=1566.402 ms
64 bytes from 10.0.0.1: icmp_seq=575 ttl=64 time=578.149 ms
64 bytes from 10.0.0.1: icmp_seq=576 ttl=64 time=8.687 ms
64 bytes from 10.0.0.1: icmp_seq=577 ttl=64 time=43.058 ms


I can login to SSH, barely. Takes more than 10 seconds and typing has like a second delay on each character, but there seem to be no CPU load and no other excessive use of RAM or processes. It looks like congestion on the interfaces but with the slowness its hard to troubleshoot.

I'll continue troubleshooting but if this sounds familiar to someone please share your insights.

Thank you.

...

apologies.

Apologies @TheHellSite.

Its only a notice:

Notice   kernel   pf: state ID collision: id: 0300000065ad36a5 creatorid: 2511b8bc

but they still appear. Aren't state ID supposed to be unique? What is causing that, any ideas?

Got it working again.

I have no idea what is / was going on but as Murphy is always around the corner, there was another issue. At one point I was able to login to the console and between all the blurp of messages I noticed igc1 (LAN) "no carrier" .... you gotta be kidding me ...  :-X

I checked and it was plugged in properly, it fits so well there's not even any play. I use good quality cables. I took it out, plugged it back in and it came up. Lo and behold ... ping replies!

I disabled the NTP and DNS NAT rules, I  think they were part of the problem.

I also reverted HAproxy config to last night and the messages seem to have disappeared.

I still don't know what is / was the problem with that in the images.

second image.

Last changes I made are with regards to HA Proxy. Last night I left everything in a working state, this morning everything worked fine. During the day I noticed these lines in the logs on the dashboard:
pf_map_addr: selected address 10.26.14.1
pf: stack key attach failed on all: UDP in wire: 10.0.0.10 185.51.192.61:123 etc (see screenshot).

This didn't seem to matter much and Im pretty sure it can be ignored? I think it has to do with a NAT rule redirecting NTP to my firewall but it doesn't work right with the wireguard interface.

Then I made changes to HA Proxy today, added home assistant which isn't working. I tried some random things and left it to walk the dog. When I came back, OPNsense was down. It still responded to the ACPI shutdown via the power button so it wasn't really dead, just unreachable. It came back up with the second screenshot:
pf: state ID collision: id: 000000blablabla creator id: 884dcb1b

I searched for it, I have no idea what it is or what it means.

Also, I have no idea if these two messages are related, and I also don't know whether they are the cause for the outage, and I also don't know if its related to HAproxy. Turns out, I don't know much of anything  ::).

So, I reverted the HA proxy changes I made today manually. It made no difference.
I turned off HAproxy, no difference.
I restored the config from last night. No difference.

I am at a loss and my internet is down  :o.

I can restore from 2 days ago (before HAproxy) but at this point I doubt that matters?

If anyone has an idea what might be going on I'd love to hear about it.

Thanks!

and another victim of this error here  :-\
both when trying to connect via

Code Select Expand

http and

Code Select Expand

https

Code Select Expand

2023-11-22T16:33:22 Informational haproxy 134.xx.xx.xx:41647 [22/Nov/2023:16:33:22.341] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:21 Informational haproxy 134.xx.xx.xx:41645 [22/Nov/2023:16:33:21.262] 1_HTTP_frontend/127.4.4.3:80: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41642 [22/Nov/2023:16:33:18.847] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header
2023-11-22T16:33:18 Informational haproxy 134.xx.xx.xx:41641 [22/Nov/2023:16:33:18.795] 1_HTTPS_frontend/127.4.4.3:443: Received something which does not look like a PROXY protocol header

Versions:

Code Select Expand

Name HAProxy
Version 2.6.15-446b02c
Release_date 2023/08/09

Code Select Expand

Versions OPNsense 23.7.8_1-amd64
FreeBSD 13.2-RELEASE-p5
OpenSSL 1.1.1w 11 Sep 2023

I ran out of ideas what to try  ???

config is:

Code Select Expand


#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    4
    hard-stop-after             60s
    no strict-limits
    maxconn                     10000
    tune.ssl.default-dh-param   8192
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local1 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    maxconn 5000
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc
    default-server maxconn 5000

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Resolver: opnsense
resolvers 64fcd546611ba3.78740961
    nameserver 127.0.0.1:53 127.0.0.1:53
    nameserver 192.168.178.1:53 192.168.178.1:53
    nameserver 9.9.9.9:53 9.9.9.9:53
    nameserver 192.168.80.2:53 192.168.80.2:53
    parse-resolv-conf
    resolve_retries 3
    timeout resolve 1s
    timeout retry 1s


# NOTE: Mailer alert bofh ignored: not configured in any backend

# Mailer: alert CB
mailers 64fcc379c27b34.94392037
    timeout mail 30s
    mailer blah.blubb.25


# Frontend: 0_SNI_frontend (Listening on 0.0.0.0:80,  0.0.0.0:443, )
frontend 0_SNI_frontend
    bind 0.0.0.0:443 name 0.0.0.0:443
    bind 0.0.0.0:80 name 0.0.0.0:80
    mode tcp
    default_backend SSL-backend

    # logging options

# Frontend: 1_HTTP_frontend (listening on 127.4.4.3:80)
frontend 1_HTTP_frontend
    bind 127.4.4.3:80 name 127.4.4.3:80 accept-proxy
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }

    # logging options
    # ACL: NoSSL_condition
    acl acl_6314a0aad6d518.84034638 ssl_fc
    # ACL: find_acme_challenge
    acl acl_6339cb3bd963e1.30823960 path_beg -i /.well-known/acme-challenge/

    # ACTION: HTTPtoHTTPS_rule
    http-request redirect scheme https code 301 if !acl_6314a0aad6d518.84034638
    # ACTION: redirect_acme_challenges
    use_backend acme_challenge_backend if acl_6339cb3bd963e1.30823960

# Frontend: 1_HTTPS_frontend (listening on 127.4.4.3:443)
frontend 1_HTTPS_frontend
    http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
    bind 127.4.4.3:443 name 127.4.4.3:443 accept-proxy ssl curves secp384r1  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6314a6a33cce38.68245567.certlist
    mode http
    option http-keep-alive
    option forwardfor
    http-request use-service prometheus-exporter if { path /metrics }
    timeout client 15m

    # logging options

    # ACTION: PUBLIC_SUBDOMAINS_map_rule
    # NOTE: actions with no ACLs/conditions will always match
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/6314a164535f16.33310179.txt)]

# Backend (DISABLED): SSL-backend-old ()

# Backend: HomeAssistant_Backend (Homeassistant)
backend HomeAssistant_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server HomeAssistant 192.168.80.21:8123 resolve-prefer ipv4

# Backend: PhotoPrism (PhotoPrism App on TrueNAS)
backend PhotoPrism
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PhotoPrism 192.168.80.30:2342

# Backend: Syncthing (Syncthing on TRueNAS)
backend Syncthing
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Syncthing 192.168.80.17:20910

# Backend: Paperless (paperless-ngx DMS)
backend Paperless
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperLess 192.168.80.30:8000

# Backend: FileBrowser (filebrowser on TrueNAS)
backend FileBrowser
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server FileBrowser 192.168.80.17:10187

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580

# Backend: SSL-backend (SSL backend pool)
backend SSL-backend
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    server SSL_server 127.4.4.3 send-proxy-v2 check-send-proxy

# Backend: Libre_photos_backend (LibrePhotos in VM)
backend Libre_photos_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server LibrePhotos 192.168.80.30:3000

# Backend: Nextcloud_Backend (Nextcloud Backend)
backend Nextcloud_Backend
    # health checking is DISABLED
    email-alert mailers 64fcc379c27b34.94392037
    email-alert from a@b.c
    email-alert to a@b.c
    email-alert level alert
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Nextcloud 192.168.80.30:80 resolve-prefer ipv4

# Backend: Jellyfin_backend (Jellyfin in VM)
backend Jellyfin_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server Jellyfin 192.168.80.30:8096

# Backend: PaperMerge (papermerge DMS)
backend PaperMerge
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    http-reuse safe
    server PaperMerge 192.168.80.17:10141



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED

frontend prometheus_exporter
   bind *:8404
   mode http
   http-request use-service prometheus-exporter if { path /metrics }


should I switch to nginx as reverse proxy ???

really?

Did you (or anyone else perhaps) ever figure this out?

I've set this up following the guide, everything works fine, but HA doesn't. I find old reddit threads and things about websockets, but it just doesn't want to work.

Thanks. I never really thought about it and I enable NAT reflection by default because at one point I actually needed it but never reconsidered why I still have enabled. Turns out I don't actually need it at all ;D

The guide I linked explains split DNS or NAT reflection is required when accessing a public service internally.

After reading your reply, I disabled NAT reflection, rebooted and removed the DNS overrides. I tested it and it resolves to public IP. The webpage still loads up, and with the new wildcard certificate that I created during the guide. It seems you are right and "it just works". Neither options are actually required.

I also just realized I can  move the services one at a time, so i'll migrate them gradually over to HA Proxy.

Have you tried reverting:
https://wiki.opnsense.org/manual/opnsense_tools.html

Will this: https://forum.opnsense.org/index.php?topic=23339.0

work properly with NAT reflection and a s2s over wireguard (between 2 opnsense firewalls)?

Story:
Before I start fiddling for hours and banging my head against the wall, I started searching for an answer. I can't figure out whether what I want will actually work. Hopefully someone can help me with an answer?

I have a whole bunch of web services, mostly running from a single docker host. Its setup with nginx-proxy for automated certificate handling. It has become increasingly more important and I need to change it to a HA setup. Furthermore, I have split DNS and NAT reflection setup. Some of these services are meant to be reachable from the outside, others are internal only.

Then some services run from a Pi or some other host, and getting them to renew certificates is cumbersome, as I have to manually disable one port forward and enable another, run the renewal and set it back.

And then yet another few services are offsite, accessible via s2s wireguard. I currently have a second nginx-proxy container running there specifically for the services running over there.

If I would setup HA proxy following that guide, it would ease my life considerably if that worked for what I need. Will that work in my setup with NAT reflection and the s2s? I would remove nginx-proxy with acme sidecar everywhere, I could use some random high ports on the docker containers and setup firewall rules to prevent hitting those services directly. All traffic would then be handled by HA proxy on OPNsense. Does it complicate things considerably compared to the guide?

These days, redirecting 53 is definitely not enough to force your DNS server. You need to block 853 out. And block DoH via (of course incomplete and ever growing) DNS blocklists. And then people use other non-standard ports for DNS.

Only when you allow all out which I think is bad practise. I block all destination ports except what I deem necessary. If a client requires a non-standard port, I open that through a separate rule for that client.

Yes, its the same on my setup, catch and redirect DNS through a NAT rule. I didn't understand what's impossible? To check if you have any leaks or that leaks should be impossible?

I meant the leaks. If the NAT rule is setup correctly, it should be impossible to get leaks to cloudflare or Opendns, or any other DNS server.

I think what might be happening is that the client sends a query to some random DNS server, the query gets redirected and a reply is sent back. The client thinks it was answered by some random DNS server while in reality, it was your opnsense replying.

I remember I looked into this in the past and was puzzled as well. I did some tcpdumps and analysis from opnsense, but never saw any traffic going out on 53. The only thing I did see was root server queries from Unbound itself which IIRC is by design. This was before I was using NextDNS and I assume its the same now.