Messages

It might just need to install the bootloader to the disk. Can you boot to a usb stick/live freebsd distro? Easier to use the native freeBSD tools not the linux ones for this case.
When booted to the live session, show the partitions on this disk.:
$gpart show

with loads of successful upgrades so far I can only assume some sort of setup/settings have come to bite.
What makes you think that you need to install the bootloader to the secondary drive? Do you have a mirrored pair of disks for the OS ? If yes, then it is a likely possibility but if you can boot after the upgrade, then you can look at that afterwards.
What's the status now, is it rolledback via a snapshot?

Note that if you had to put the isp router back in and then had router-behind-a-router, then that could account for no clients hitting it.

@cookiemonster was there a time when Unbound in OPNsense didn't support multiple DoT resolvers?  I don't remember but I'm curious what stubby is adding in that chain.

It probably didn't support multiple DoT resolvers at the time. I set it up some years ago and since multiple OPN upgrades so it has stayed. Maybe it was also some additional options that made it more desirable at the time: options of distribution to upstreams, ciphersuites, and other available knobs to tweak. I fail to remember.
Now looking at Unbound, if simply having DoT with multiple upstreams is the only wanted feature then you are right, Unbound on it's own would do.
The setup has served me well all along so I'm happy to leave it as it is. Good point though.

I still use stubby on OPNSense which I think I documented sometime here. However it was not to have logic to direct queries in different directions. In fact that is news to me that it can do that. Mine use has clients directed to AdgH , which then has Unbound as the upstream resolver. Then Unbound forwards ALL queries to Stubby. Stubby is configured with a few DoT upstream resolvers in round-robin fashion.
In other words the dns selection for clients if I wanted would go on AdGH as Patrick says.

Problem here is that we can't guess your internal setup. You talk about routing but all "internal" traffic i.e. devices talking to each other in your "LAN", they don't go through your OPN router. They talk via a switch that you will have downstream of OPN and plugged into that LAN port.
That said, it could be routing if you have more than one network segment and traffic has to go through your OPN, but this is what we can't guess. You must give us all the details.
Finally if resetting that interface seems to make a difference, that suggests the router is involved somehow. Again tell us the setup.
And hello to  you too.

All good

@(MARLOO)if you do

Code Select Expand

#zpool status you should get a message like:

Code Select Expand

  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.If so, then it would match what dmesg spits out. It's meant to say that as a ZFS pool there are features available but not used until a zpool upgrade is made. Just as the messages say, they are but are up to de administrator to upgrade the pool if they want to use them.
Please please do NOT blindly do a zpool upgrade unless you know what you are doing and NEED those features. Just like the message says.
It can require the booloader to be upgraded as well or you have a non-bootable machine.
See the zpool-features(7) for those details.
To read them: $man zpool-features.

this isn't an known issue. Either configuration problem, hardware problem or both.
You won't have answers without expressing your setup, hardware, etc. All you would expect anyone bringing to you to diagnose a problem with something you are familiar with.

I couldn't find the conversation on this forum in my mind.
So in the meantime I'll still suggest either testing the available api or do this unix-to-human change/append to the file, either before sending or at the destination. Sure both have pros and cons.

It would be for anyone. You're not wrong.
The system is unix-like so it has a date function. If you type "date" without the quotes, it'll present the human-readable date and time.
You could write some script to append it or something like that.
Let me see if I can find the previous conversation on this forum about it.
p.s. the download will also have a human timestamp of the downloaded file on the receiving system, which could also be useful.

I'm not sure this is a case where the rule should be using direction OUT.
Can you try this rule but direction IN, and have a ready-made pass for where the traffic will flow into the firewall. It seems for first post you are sending it to loopback so I expect not needed but I might not see the whole picture.

but why limit your investigation to that metric only?
Unless I misread the whole thing you have a firewall that has an apparent overuse of cpu cycles. You are trying to investigate what _process_ might be the one(s) causing the spike in cpu cycles. Here is what my suggestion is, to use htop to see what processes are using up the cpus. From that you can hopefully identify the processes and start digging into them to figure out why.
A ps no matter what flags to use is only a point-in-time list of processes. top/htop are updating real time unless the whole machine is too overwhlemed.

Just need to figure out what is the actual problem. use your browser's developer tools to see what is the actual error if there is one.

It sounds like it's configuration problem.
When you go from VM to baremetal or viceversa, there are changes to be made. There is no proxmox bridge anymore, the firewall rules will need to be adjusted for instance.
Either will work fine on small networks until you need high cpu performance for IPS for instance.
My main OPN (at home) runs as VM on a small Chanwang thingie and has spare resources. It uses an AMD Ryzen 5 5600U with just 2 cores, 2 threads. Uses various services, including Zenarmor, a few VLANs. No problem.
In sum, just review or post your settings and we'll help you figure out the problem. Doubt is performance related.

You can just invoke "top" and sort by CPU. FreeBSD top can do that. And lots of other things.

True, just preferences. I find htop better shows me the process tree and colours. Just personal preferences.

Could you install htop, launch with admin privs and sort by CPU ? May show a clue

You can just invoke "top" and sort by CPU. FreeBSD top can do that. And lots of other things.

done in the other thread i link, the top shows a 100% usage in the interrupt level specially in the netisr irq handler, see the picture:
https://forum.opnsense.org/index.php?action=dlattach;attach=56143

Sure but I'm referring to htop allows to see the process tree in real time which is of more interest.