Messages

This has nothing to do with Proxmox.

If you want Unbound to resolve local hostnames, you will most likely have the domain osx.ninja forwarded to Dnsmasq on port 53053.
You must do the same for "1.168.192.in-addr.arpa" to enable reverse lookups.

The rules look "O.K." now. I told you to first check if your rules cause the problems in my first answer, for somehow I guessed that you redirected "all" port 53 traffic, creating an endless loop like explained here.

What can you learn of this? Your rule of thumb should be: If you experience problems, show your rules, because often times, they are the cause of it. See also the "READ THIS FIRST" article in the tutorial section.

That being said, your current rules alone will not help you with either DoT or DoH, which are the default in many browsers now.
There is a discussion about this also in the tutorial section.

Basically, you can block port 853 for DoT, but you need a blocklist for known DoH services because you cannot block port 443.

Also, there are a few more kinks in your rules, because they apply to IPv6 as well, see this.

On a side note: I have given up on the "block any other DNS than my own" game, because you cannot win it.

Maybe there will eventually be a fix for this: https://github.com/freebsd/freebsd-src/pull/2318

I proposed to ,,try" disabling These rules in the absence of you having presented your exact rules, which in Turn prevents anyone from inspecting them for any potential errors.

If Unbound works without them being enabled, it would prove something is wrong and then we can start looking. So basically, this is just a quick and Dirty Test, not a permanent Solution.

Both could be blocking Unbound Traffic when slightly wrong.

Try disabling the firewall rules for a try. It might be the case that you created a loop where Unbound tries to query itself when it makes upstream queries.

Why not using good, old IPv4 for internal things?

I am also fond of that, see: https://forum.opnsense.org/index.php?topic=45822.0 for an in-depth discussion.

Correct.

I mean that it is probably the CPU waiting for I/O operations to finish because the storage subsystem is struggling. With NVMe storage, writing very small, frequent log lines leads to heavy 'write amplification' due to the large underlying flash block sizes, causing significant latency spikes.

The fact that reducing the logging activity resolved the issue completely confirms the storage bottleneck theory. When the logging was set to high, the frequent, small writes to the NVMe caused massive write amplification and continuous synchronous I/O blocks.

In FreeBSD/OPNsense, such a severe storage backlog causes 'Lock Contention' in the kernel and forces an influx of Software Interrupts (SWIs) to manage the blocked I/O buffers. This is why the IRQ handler threads (intr) ran hot: they weren't just processing network packets, but handling the massive overhead of a stalling filesystem trying to commit log rows to a struggling SSD. Turning down the logs broke this vicious cycle.

I would guess this really is more of a storage limitation because of RMW-cycles induced by ZFS.

That CPU is not the fastest, either. For each RMW cycle, it must calculate ZFS checksums. A proper centralized log server could probably buffer log messages more efficiently. If you have Monit running on top, each new message is also parsed and compared against triggers. You could use top and see which processes actually peg the CPU.

It only works for one specific monitoring IP, yes, not multiple destinations per gateway. There is no dynamic routing depending on "what is best".

On the other hand, that would also break your connections, because the routes (and IPs) would change. Normally, you would want "Failover States" to be enabled, too. If you want some destinations to use another (group of) interface(s), you can do PBR based on that.

You can use gateway groups for that and use specific trigger levels - it is in the official docs.

A system like that can be had from Quotom (Q20322G9) with passive cooling, plus it has 4x SFP+ ports on top of 5x 2.5 GbE ports instaed of 5x 1 GbE.

Das ist nicht wahr: Ich sehe "Hetzner DNS" und "Hetzner DNS Legacy (deprecated)" als Service in der GUI angeboten (26.1.11_5).

Do you guys use os-realtek-re or the native FreeBSD driver? You know that Realtek adapters are buggy under FreeBSD, don't you?