Hi, as you can see from my "activity" I didn't have a lot of problems with OPNSense. Except now when upgrading to 26.1.3, the installation had an error but it seemed to continue. After reboot, got a boot loop with text just scrolling fast on the screen. I tried loading a new kernel = nothing. I tried restoring a snapshot = nothing only another error running on the screen. That was an ugly surprise, I am used to snapshots restoring stuff in ZFS.
So I reinstalled figuring "hey should be easy to restore 'cause you have backups". But no, had to do a full install, juggling 3 cables in 4 nics (because nothing is displayed when connecting something, to see what interface becomes damn active), got on the default IP, restored. Again - no, you dont't have all the cables properly plugged - so I plugged them - oh no, you lost connectivity, so on. The circus around a restore is quite maddening.
After several reboots, I restored... something and everything seemed in place. Yes! But again - no: I don't have the plugins. So I went to System > Firmware > Plugins, and they were in red all right. Tried to install, but:
***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 26.1.2_5 (amd64) at Sun Mar 8 12:42:28 EET 2026
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 26.1.1 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 26.1.1 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
No plugins found.
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 26.1.2_5 has 67 dependencies to check.
Checking packages: .......
dnsmasq-2.92,1 version mismatch, expected 2.92_2,1
Checking packages: ................
opnsense-26.1.2_5 version mismatch, expected 26.1.3
Checking packages: .
opnsense-installer-25.1_1 version mismatch, expected 25.1_2
Checking packages: ..
opnsense-update-26.1.1_1 version mismatch, expected 26.1.3
Checking packages: ......................
py311-Jinja2-3.1.6 has no upstream equivalent
Checking packages: .
py311-dnspython-2.8.0_1,1 has no upstream equivalent
Checking packages: .
py311-duckdb-1.4.4 has no upstream equivalent
Checking packages: .
py311-jq-1.11.0 has no upstream equivalent
Checking packages: .
py311-ldap3-2.9.1_1 has no upstream equivalent
Checking packages: .
py311-numpy-1.26.4_12,1 has no upstream equivalent
Checking packages: .
py311-pandas-2.3.3,1 has no upstream equivalent
Checking packages: .
py311-requests-2.32.5 has no upstream equivalent
Checking packages: .
py311-sqlite3-3.11.14_11 has no upstream equivalent
Checking packages: .
py311-ujson-5.11.0 has no upstream equivalent
Checking packages: .
py311-vici-6.0.3 has no upstream equivalent
Checking packages: .......... done
***DONE***
So now I am supposed to get to 26.1.3 to have plugins?
Sorry to sound bitter, but I am not a networking expert so it took me a long time to have 13 VLANS, NGINX proxy, adblockers, blacklists, dual wans, policies and lots of firewall rules.
I restored a bunch of other OSes, switches, VMs, NASes, and even pfsense that I was using before, quite a lot of stuff. Never before I was more perplexed at restoring something, without internet of course - so if you guys could something to improve on that, to streamline it and so on - it would be great.
Now, what can I do about messages like these, when trying to install NGINX, without having to again bork the system with a 26.1.3 upgrade?
***GOT REQUEST TO INSTALL***
Currently running OPNsense 26.1.2_5 (amd64) at Sun Mar 8 13:03:44 EET 2026
Installation out of date. The update to opnsense-26.1.3 is required.
***DONE***
Thank you, and please allow us to vent a bit here - this became a very ugly week-end, for quite a few people it seems.
PS: I did look for a 26.1.3 image, but it doesn't seem to exist, so no fresh install there. And the complete list of missing plugins is:
os-acme-client (missing) 4.14 814KiB 3 OPNsense ACME Client
os-dmidecode (missing) 1.2 6.71KiB 3 OPNsense Display hardware information on the dashboard
os-isc-dhcp (missing) 1.0_4 280KiB 2 OPNsense ISC DHCPv4/v6 server
os-netdata (missing) 1.2_1 17.2KiB 3 OPNsense Real-time performance monitoring
os-nginx (missing) 1.36 964KiB 3 OPNsense Nginx HTTP server and reverse proxy
os-nut (missing) 1.9_1 55.4KiB 3 OPNsense Network UPS Tools
os-sftp-backup (missing) 1.1_2 13.6KiB 2 OPNsense Backup configurations using SFTP
os-smart (missing) 2.4 22.9KiB 3 OPNsense SMART tools
os-udpbroadcastrelay (missing) 1.0_6 43.9KiB 3 OPNsense Control udpbroadcastrelay processes
os-wol (missing) 2.5_3 22.5KiB 3 OPNsense Wake on LAN Service
You're probably going to need a reinstall. If there's no direct 26.1.3 image, an upgrade from the latest version available should hopefully not go the same way.
Stupid question, did you try re-running the update process?
But I did reinstall, and restore the xml, as stated - I am on the image that is available for download 26.1.2_5
Strange enough the 26.1.3 is still available as an upgrade in the interface. I didn't try it again after the reinstall+restore, because of the problems - seemed silly to repeat them.
As it is proposed as an update only, and given the problems (not as an image for download), figured I should wait, maybe is in the process of being dropped because of the widespread issue or something? Better be, I already f... hate it :)
So: initially I was on the 26.1.1_something carried after years of upgrades, and then I tried to upgrade to 26.1.3 - that's when things have gone berserk. After it was clear that the system will not boot, I reinstalled with 26.1.2_5 and restored settings from a 26.1.1 backup. And I was left with no plugins.
Tell me , friends, did you got Segmantation Fault red popup ?
Yes
>>>>> So now I am supposed to get to 26.1.3 to have plugins
In not so many words, Yes.
Simply check for updates, from CLI or GUI, wait for the system to reboot. When back up do another check just in case. If unsure you can post here the full output.
Your "nightmare" situation is not unique to 26.1.3, can happen on any upgrade when on ZFS and running out of space. It is an ugly scenario that you can easily reproduce in a vm: fresh install, take snapshots, fill the disk, start an upgrade.
@newsense please do not take lightly the hurdles of the community. We are not ganging here to jump the team or something, nor we are all devs or network professionals, quite clear from the lack of info and the impossibility to even boot, to offer you more info. So there is no Me2s statements, only confirmations of the number of occurrences.
And when someone puts down their story in so many words, explaining what have they done and how it went wrong, and how it is still "wrong", your answer starts with "simply..."?
Didn't I just tried the "simply" or what? Does it make any difference that for me happened from from 26.1.1 and now it is supposed to simply work from 26.1.2? When pretty much all the other users reporting this were on 26.1.2?
If you don't intend to be helpful and give us some insights, for example why snapshots can't do stuff, wait for a release, we are investigating, try this or that, please refrain from shaming your users as I am too old for this old open source story.
"Nightmare" – are you for real? You realize this problem results in no network&internet access. Or at least stuck with no other services as explained.
Here, are some screenshots of the errors I took right after upgrade and the 1st reboot. These were the messages continuously spammed on the screen. Maybe they will help, though I don't think they will.
https://pasteboard.co/JV4GDUgU2rMK.png
https://pasteboard.co/iGs1f8BACGzt.png
https://pasteboard.co/xIFvnHri5lin.png
I gave you all the steps on how to fix your new install and how to reproduce the "nightmare"
>>>>> why snapshots can't do stuff
ZFS has no provisions for snarky users when running out of space. When that happens *everything* is lost.
The only apparent issue with 26.1.3 based on your initial description is your refusal to use the proper steps to fully upgrade your new installation.
I am sorry - you don't make any... sense. How did you come up with the conclusion that there wasn't any space left on the device, when it was installed on a 250GB drive with only a fraction used? And the like 3 snapshots were around 1, max 2GB each?
So let's leave it like that, maybe someone else will solve it, have a release or an advice for us, I will simply ignore your posts.
Quote from: fakemoth on March 08, 2026, 07:03:30 PMI am sorry - you don't make any... sense. How did you come up with the conclusion that there wasn't any space left on the device, when it was installed on a 250GB drive with only a fraction used? And the like 3 snapshots were around 1, max 2GB each?
So let's leave it like that, maybe someone else will solve it, have a release or an advice for us, I will simply ignore your posts.
I've been through your exact nightmare scenario twice since December. One was a physical FW where there were lots of snapshots, the other a vm with plenty of space available but the underlying storage partition.
For all the good things ZFS does the one thing it wasn't designed for is to deal with space running out.
There _is_ a slight chance your ssd might be partially broken, which is usually the hardest to diagnose because all seems to work until it doesn't, however the fact you've been able to reinstall makes me think you're not there.
Take a snapshot, start upgrading, post here any updates or things that appear to be happening and you're not sure what to do about. This is the only constructive thing you can do right now. There are no magical fixes coming.
P.S I'm just a community member in case you're wondering. I've been running what is now 26.1.3 a week before it was released on multiple firewalls across the globe. An issue resulting in total loss of data would have been reported long before it was published.
On top of that, none of the kernel patches touched ZFS. And assuming the software packages would have resulted in a system stalling on boot somewhere any and all your snapshots would have been available for you to use.
Does it mean it's random in your eyes ?
There is no known and wide spread problem that will nuke running OPNsense systems when upgrading to 26.1.3.
I run more than a dozen OPNsense systems and while admittedly I have not updated every single one to 26.1.3 yet, for the ones I did I had no problem whatsoever.
Maybe @newsense was a bit less diplomatic as desired but in general the folks helping in this forum are engineers, not diplomats. And that includes myself. I have my own history of being rather blunt, but please assume that I never intended anything ill.
If I read this thread correctly, you had an unsuccessful update to 26.1.3. Well, that happens, occasionally.
But @newsense was perfectly correct in pointing out that the most reasonable way to a running system is:
- install 26.1 from a downloaded image
- restore your saved configuration
- update to 26.1.3 - yes!
- let the system install all missing plugins automatically
That's just how it's supposed to work.
The big argument in this is thread seems to be:
- You - there is some fundamental bug in 26.1.3 and it will break any time I try an update again.
- Everybody else - no, there isn't, and while you had an update fail in a bad way, there is no reason to assume it will fail again on the next try.
There is no known issue with 26.1.3 that will break "Internet". Whatever happened at your first try is particular to your single system. Please just try again from a fresh installation, and the builtin wizard will automatically install all missing plugins. That's how it is supposed to work. See above for the steps I outlined.
There is no mechanism to install plugins for a version that isn't current. Install base, update to latest, then fix plugins. Always worked that way.
Kind regards,
Patrick
Quote from: DiskWizard001 on March 09, 2026, 12:20:45 AMDoes it mean it's random in your eyes ?
It is not random.
It is not a OPNsense issue.
I explained why it happened and how to consistently trigger it if you wish in a vm.
The way to think about this issue to avoid other accidents in the future is this:
QuoteDo not run out of space while the ZFS file system is modified during OS updates
If you would have been lucky enough to run out of space while the updates were downloading you would have been able to recover and avoid total data loss. Once the updates are being installed you're past the point of no return.
Quote from: newsense on March 08, 2026, 07:36:03 PMThere _is_ a slight chance your ssd might be partially broken, which is usually the hardest to diagnose because all seems to work until it doesn't, however the fact you've been able to reinstall makes me think you're not there.
Granted it's not a proper SSD, but with the thumbdrive I'm using for testing I had a similar failure that would throw me into the single user console on boot, after random errors each subsequent boot (it was not even updating, just booting). I pushed nano back onto it and it booted and upgraded normally (at one point I got the red warning but it vanished mysteriously so I assume it was from upgrading base). Anyway, I know that on this drive there are some cells that can't hold their stuff for more than some weeks but turn up fine when freshly written. A proper SSD should catch this but just like with hdds it may flag them as "maybe unstable" but fail to actually reassign them (had this myself), leaving the issues in place. Examining smartctl -a might give insights into the actual state of degradation of the drive.
Quote from: newsense on March 09, 2026, 02:02:54 AMThe way to think about this issue to avoid other accidents in the future is this:
QuoteDo not run out of space while the ZFS file system is modified during OS updates
Is ZFS somehow worse in this respect than others? If you run out of space while upgrading Linux on ext4, chances are you're stuck with an unbootable system, too, except in Linux you normally have at least one fallback kernel, and also can usually install packages from at least one prior major version. I've read that ZFS doesn't like getting filled but it only becomes slow if it does, not inherently lose data?
Once it's 100% full, a ZFS pool is broken, because you cannot even delete things, anymore. It's copy on write so to delete you need space to write first.
Thanks, that is an important fact! Feels like an odd design decision, (more like a major flaw), especially when used as system FS. The lack of recovery tools for ZFS has always made me wary of it, plus that it is complete overkill for a simple system FS. Probably OK for a NAS where you can control the pool usage, but not on a desktop. Boot environments are nice but this... nah. :) Maybe the idea is to add another device to the pool, it should be able to do that even if the pool is 100% full since the journal would not be touched by that?
The -nano image is so much more convenient to use, anyway, big thanks to the devs for making it, doubly appreciated! :)
Monitor your usage? Network management systems and SNMP or telegraf and prometheus or Zabixx or ... exist.
Yeah, on the OPN box Monit is set to trigger on 75%. These devices should never run low on space in the first place. Monitoring in general is a good idea, but I still prefer recoverable faults.
Everything that requires discipline to keep working is going to break. :)
Quote from: drosophila on March 09, 2026, 02:50:02 AMIs ZFS somehow worse in this respect than others?
If you run out of space while upgrading Linux on ext4, chances are you're stuck with an unbootable system, too
Quote from: Patrick M. Hausen on March 09, 2026, 07:34:05 AMOnce it's 100% full, a ZFS pool is broken, because you cannot even delete things, anymore. It's copy on write so to delete you need space to write first.
I am confused here : Both OpenZFS and EXT4 for example have Reserved Space that prevents them from getting completely full AFAIK ?!
When you install a random Linux distro it shows 5% Reserved space for EXT4 and can be set back to 1% for large drives/partitions.
For OpenZFS there is this setting : https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#spa-slop-shift
To check your current setting :
# cat /sys/module/zfs/parameters/spa_slop_shiftSoo...
Right or Wrong ?! o.O
Quote from: nero355 on March 09, 2026, 10:16:08 PMI am confused here : Both OpenZFS and EXT4 for example have Reserved Space that prevents them from getting completely full AFAIK ?!
Not to my knowledge. We had multiple incidents of unrepairable pools over in TrueNAS land. Once it's 100% full it's toast.
Quote from: Patrick M. Hausen on March 09, 2026, 10:19:40 PMWe had multiple incidents of unrepairable pools over in TrueNAS land. Once it's 100% full it's toast.
IMHO that should not be possible unless someone did some very weird things with the value mentioned above and set it to 0 somehow ??
What is the default value for OPNsense installs when OpenZFS is chosen ?
/sys/module/zfs/... does not exist on FreeBSD.
https://forums.truenas.com/t/zfs-pool-ko-after-filling-at-100/57356/9
Thank you. I just took the advice there and reserved 250GiB for the root dataset and children in TrueNAS.
The reserved 5% on Linux ext4 is for the system so that ordinary users cannot fill up the drive and thus the OS can still operate and root can install stuff. The problem is that updates always are done by root (who is fully entitled to these 5%), so if root fills these up, be it through an update or otherwise, they're gone. On ext4 this isn't for the FS to remain usable, ext doesn't run into any issue when it fills up. ZFS is the only one I've ever been made aware of having this odd problem. Obviously its benefits outweigh these shortcomings, at least for corporate storage servers. On a data pool, this is easily avoided by having only normal users use it for storage (remote root access always is bad even without this). At least my (XigmaNAS) pool was perfectly fine when I "ran out of space" on it (partial write, reporting "no space left on device"), but it was non-root users only, so the emergency reserve would have been untouched. Obviously, this cannot be avoided with RootOnZFS. It still is odd that, especially given how wasteful ZFS is, it wouldn't just keep a minimum of spare space to itself regardless of who is accessing it so that at least deletes could still be done. Then again, there are so many tunables that might have unintended side-effects which may look like optimizations but end up creating such situations, whereas the defaults are fine?