Hi,
Just a heads up that when I upgraded this evening to the latest 25.1.5 the system was broken on reboot. Don't exactly know how it is broken but Nothing on my home network is working. I suspect that somehow the DHCP is the issue, but don't know for certain. Cannot access the webportal but when I plug a monitor and keyboard into the router everything looks OK and the main WAN has an IP and my Wireguard tunnel is up.
Fortunately I have a full disk backup from Sunday so when I can remember how to uncompress it and dd it back to the main disk I should be somewhat OK, but that will have to wait for tomorrow.
Stephen
Captive portal running? See https://forum.opnsense.org/index.php?topic=46775.0
Cheers,
Franco
I never setup captive portal.
How would I go about disabling captive portal from the command line - just to be sure.
Just upgraded to 25.1.5 and it didn't reboot. Hooked up a monitor to see what was going on, and it can't mount root: unknown filesystem.
I took a snapshot before the upgrade, as I always do, but I can't see the usual menu that allows me to rollback to a chosen snapshot.
I'm at a mountroot> prompt, I guess it's expecting I specify a filesystem, I used zfs, but none of the ones I tried worked.
Any suggestion??
Quote from: alexdelprete on April 10, 2025, 11:54:29 PMJust upgraded to 25.1.5 and it didn't reboot. Hooked up a monitor to see what was going on, and it can't mount root: unknown filesystem.
I took a snapshot before the upgrade, as I always do, but I can't see the usual menu that allows me to rollback to a chosen snapshot.
I'm at a mountroot> prompt, I guess it's expecting I specify a filesystem, I used zfs, but none of the ones I tried worked.
Any suggestion??
Drive full or dying most likely. Are you sure you're not skipping over the boot menu ? if that appears you could try booting the old kernel - just in case the new kernel wasn't installed properly.
Try a fresh install, see how the drive behaves.
Quote from: newsense on April 11, 2025, 12:30:12 AMDrive full or dying most likely. Are you sure you're not skipping over the boot menu ? if that appears you could try booting the old kernel - just in case the new kernel wasn't installed properly.
Try a fresh install, see how the drive behaves.
It's a 1TB nvme drive, 99% free. Never had issues with it. The boot menu doesn't come up, I see a strange booting /boot/kernel/kernel text line with some hex characters. I managed to press space to get to an OK prompt in which I have some commands available, but I don't know how to load the old kernel from there.
(https://i.ibb.co/bgz6grmN/Phone-Experience-Host-JJip-SXb-MFW.png) (https://ibb.co/ynWNn8Pg)
If I don't do anything and it loads the new kernel, then it stops here:
(https://i.ibb.co/ZR8JbyhL/Phone-Experience-Host-nnfr-D7l-Bo6.jpg) (https://ibb.co/RG0SJLCv)
I guess I'm stuck and have to reinstall, right?
Yes that's the path forward right now, the drive is still the unknown here.
Let's see what's going on, hopefully nothing unresolvable.
Can you please boot with a liveusb (not linux, we need freebsd). Best to use the same freeBSD version in case of need of the boot code.
Boot with it and drop to a shell and issue a $gpart show and provide the results inside code brackets.
I reinstalled from scratch and restored the config manually (I had git backup and also manual backups of the config). I double checked the nvme drive and it has no issues I can diagnose. This means that something happened during the upgrade. :(
First time in years I had issues with an opnsense upgrade. Must confess that now I'm a little bit scared for next upgrades.
After this experience, What I feel is missing is that in the live usb image there is no recovery tool that checks (and fixes) the disk installation when facing these kind of boot issues.
The other pain was the fact that we have a config backup, but the plugins (and their config/data) are not restored. Now I'm back on track, almost, but some plugins I still have to configure them. Tailscale for some reason is not behaving properly, but I'll check later, will probably reinstall it from scratch.
Question is: to prevent this from happening in the future, and shorten the restore cycle, what should I do? take a full image of the drive by pulling it out of the system every once in a while? isn't there a better way to achieve this?
Quote from: alexdelprete on April 11, 2025, 01:43:11 PMAfter this experience, What I feel is missing is that in the live usb image there is no recovery tool that checks (and fixes) the disk installation when facing these kind of boot issues.
There are no offline analysis and repair tools for ZFS.
Quote from: alexdelprete on April 11, 2025, 01:43:11 PMThe other pain was the fact that we have a config backup, but the plugins (and their config/data) are not restored. Now I'm back on track, almost, but some plugins I still have to configure them. Tailscale for some reason is not behaving properly, but I'll check later, will probably reinstall it from scratch.
Question is: to prevent this from happening in the future, and shorten the restore cycle, what should I do? take a full image of the drive by pulling it out of the system every once in a while? isn't there a better way to achieve this?
Here is the pitfall of modifying outside the UI which acts as a sort of collector of the modifications for reinstallations. Also shows the advantage of running it as a virtual machine.
Enven then we have to backup the image of the hypervisor somehow, like taking a full image of it. Or, what takes care of it in both cases is to run it on high availability storage i.e. a raid setup. Even a mirrored pair pretty much takes care of it BUT it is of course sometimes not possible like when not available storage ports.
Reminds, me. I need to make a new image too but has downtime. Boot to Clonezilla, clone to extenal disk.
Quote from: Patrick M. Hausen on April 11, 2025, 01:59:27 PMQuote from: alexdelprete on April 11, 2025, 01:43:11 PMAfter this experience, What I feel is missing is that in the live usb image there is no recovery tool that checks (and fixes) the disk installation when facing these kind of boot issues.
There are no offline analysis and repair tools for ZFS.
I feared (but kind of expecting it) this feedback was coming. Thanks Patrick.
Quote from: cookiemonster on April 11, 2025, 02:37:36 PMHere is the pitfall of modifying outside the UI which acts as a sort of collector of the modifications for reinstallations. Also shows the advantage of running it as a virtual machine.
Enven then we have to backup the image of the hypervisor somehow, like taking a full image of it. Or, what takes care of it in both cases is to run it on high availability storage i.e. a raid setup. Even a mirrored pair pretty much takes care of it BUT it is of course sometimes not possible like when not available storage ports.
Reminds, me. I need to make a new image too but has downtime. Boot to Clonezilla, clone to extenal disk.
HA storage doesn't solve the issue of an upgrade script creating issue, or an "rm -rf" on the wrong path. :)
But you have a point that will make me think in the next days: maybe it's time to seriously consider virtualizing OPNsense, I was not in favor of it for several reasons, but considering what happened, probably the advantages outweigh the disadvantages. The ability to quickly restore a VM, in seconds, vs spending a whole night trying to recover a bare metal installation is really tempting. Thanks for the advice.
Well my recovery image is corrupted in some way so that path is closed to me.
Going to try a factory reset and then a restore to see if it flushes out the errors. I. My case I definitely think it's the DHCP which is the root cause. The firewall itself is able to ping out and all interfaces are up and running - but no services on the LAN are getting IPs. A clue is that my Kodi boxes ran on for around half an hour before they died which sort of points to the DHCP dropping the connections at refresh. Cannot ping anything on the LAN.
Mighty pain in the hole since I have quite a few add-ons to reconfigure if I fresh install, will take the best part of a day but at least I have notes.
Quote from: Alessandro Del Prete on April 11, 2025, 02:56:36 PMHA storage doesn't solve the issue of an upgrade script creating issue, or an "rm -rf" on the wrong path. :)
Yes, true of course.
@Shoog you seem to be in a better place. Something in config only not right, not the whole OS failing to boot.
Have you installed the latest hotfix?
Quote from: cookiemonster on April 11, 2025, 03:38:41 PM@Shoog you seem to be in a better place. Something in config only not right, not the whole OS failing to boot.
Have you installed the latest hotfix?
I tried using the console upgraded but it seems to time out and then freeze.
I tried the captive portal fix which seemed to run but made no difference. Identifing the root cause is the tricky bit to resolving what's going on here.
Quote from: Alessandro Del Prete on April 11, 2025, 01:43:11 PMThe other pain was the fact that we have a config backup, but the plugins (and their config/data) are not restored. Now I'm back on track, almost, but some plugins I still have to configure them. Tailscale for some reason is not behaving properly, but I'll check later, will probably reinstall it from scratch.
Question is: to prevent this from happening in the future, and shorten the restore cycle, what should I do? take a full image of the drive by pulling it out of the system every once in a while? isn't there a better way to achieve this?
For the official OPNsense plugins all you need to do is to
import the config file - check for updates - reboot. The plugins will all be downloaded for you and upon reboot the FW will be working as expected with no additional configuration required.
For third party repos you'll need the configuration file of that plugin you're using, as it is not saved in the OPNsense one.
That's what I was referring to. I don't use a lot of plugins, but some of them are not "official", so there's no consistent way of having them backed up.
I hope one day we'll be able to have one "backup config" file/archive with all config files of all plugins, official and unofficial, that gets imported easily.
Another annoying thing was having to reformat a usb key with fat32 specifically to reimport the config.xml I had on git. I had it on my Ventoy USB drive, that is exFat, used to reinstall opnsense iso, but it didn't mount it, I had to prepare another one specifically, after trying to understand why it didn't mount it.
Little things, that all summed up make a difference when you're anxiously trying to recover the fw installation.
QuoteAnother annoying thing was having to reformat a usb key with fat32 specifically to reimport the config.xml I had on git. I had it on my Ventoy USB drive, that is exFat, used to reinstall opnsense iso, but it didn't mount it, I had to prepare another one specifically, after trying to understand why it didn't mount it.
I think I got bit by this sometime in the past. I've asked the question on behalf of users on github now as a feature request (core/issues/8538).
So things just got considerably worse for me. It seems that the upgrade has somehow trashed my GPT partitions. I performed a factory reset and now my system won't boot and the bios is no longer accessible.
Going to see if I can get it to boot from a USB, but this has gotten mighty annoying.
Quote from: Shoog on April 12, 2025, 01:30:18 PMIt seems that the upgrade has somehow trashed my GPT partitions.
That seems more like a hardware problem. Maybe your PSU or RAM is iffy?
It seems highly improbable that losing my bios and the issue with a failed upgrade are unrelated. The event that triggered a loss of bios was the factory reset. Before the factory reset the bios was accessable. It boots up to the point where it shows the bios for my Ethernet card which is itself accessable - but then as soon as it gets to what would be the normal bios and hand off to the OS it simply shuts down.
EDIT: Turned out that the factory reset re-enabled a case switch. Have bypassed it now and have been able to boot again.
I think I have got to the root of my issue. The latest update has an incompatible GPT partion for my Fujitsu S920.
I got a version of OPNsense 24.7 running on one disk drive and restored a backup config.xml, works with a few issues with my VPN on wire guard yet to be resolved, probably due to this been an effective rollback from the version I was previously running.
However I then decided to setup the latest version on a separate SSD to see if I could get it up to date. Will not boot at all from this disk complaining about GPT format, throws a hexadecimal table and freezes.
I had noticed some errors regarding GPT before but didn't pay it enough attention.
So unless I can work out why it doesn't like the GPT I am stuck.
You can do a FreeBSD installation with MBR, then use the bootstrap method to turn it into an OPNsense.
MBR works only for limited disk size - beyond that you need GPT.
But I think that many posts on the internet show the S920 BIOS to be broken (https://www.reddit.com/r/OPNsenseFirewall/comments/10a69d6/uefi_on_fujitsu_s920/) w/r to UEFI booting, so you might be better off to try to set BIOS boot.
You could also try this method (https://forum.netgate.com/topic/154769/pfsense-does-not-boot-after-successful-installation/7).
Thanks for the pointers.
Going to concentrate on getting the working install fully back up and running first. Would like to get it back to 25.1.4 where it was before but can't find clear instructions on how to update to a specific version.
Will then go back and see if I can get the other version working on a separate disk. Funny thing is that this just came out of the blue after successful update through from 24.7
Got everything up and running again after some funky side tracks along the way to complete reinstalling.
I managed to get it to upgrade all the way to 25.1.5 in the end - so the issues with GPT (which were the root cause of the upgrade fail) seem to have crept in somewhere over the last month or so of uptime. GPT is twitchy and any issues seem to proliferate across multiple HDD and even to the install USB.
In the end I reformatted the SSD I was installing to to FAT32, and reformatted and reinstalled the installer USB.
The issue is whats the best backup strategy since what I thought was adequate completely let me down (which was a Gzipped DD copy of the whole OS which obviously just copied the GPT corruption that had occurred).
Anyway alls good that ends well.
I'm having issues after upgrading to the 25.1.5_4 as well it updated rebooted and now just freezes at the main opnsens menu screen.
Is there a place to download the 25.1.4 iso it ran just fine?
Please open a new thread for your issue
Quote from: newsense on April 14, 2025, 12:50:02 AMPlease open a new thread for your issue
Sorry I figured out my issue I had to change boot to UEFI