backup story

[gratis]

This afternoon there was a brief power outage. The old UPS that was powering the OPNsense machine, along with a few network devices, immediately powered down. It appears that the old UPS needs a new battery, and a new battery status indicator light...

I installed a new APC UPS and rebooted OPNsense, but the only service that was running was ntpd. All other services, dhcpd, dnsmasq, apinger, and a few others like Proxy were Stopped, and would not Start. The log files of the services showed unusual error messages like missing user accounts, or misconfigured settings, although before the power failure everything was working fine.

It quickly became clear that the power outage had corrupted some files. Rather than troubleshooting a lot of random issues, I decided to simply reinstall and start fresh. Unfortunately I didn't have a backup of the configuration, but it's easy to recreate...

So, that is the story, for what it's worth. The question is, would it be possible to add an APC UPS monitoring package to the OPNsense repository?  https://freshports.org/sysutils/apcupsd/

[franco]

Hi gratis,

thanks for sharing this. I read about this happening on pfSense/FreeBSD, but I am unsure how to address this properly. Setting the disk to "sync" mode to force flushes all the time is something that can be done locally, but is not for a standard installation and may still fail sometimes.

In short this is what happens: users and groups added to the system disappear during the crash, the system is unable to put them back.

If you have enough of the system left in a running state (no essential libraries or binaries missing, root login working), try:

# opnsense-update -f && reboot

This will reapply the kernel and base system as well as packages for 15.1.12 systems. It might not be enough, but it's worth a try. If not it may be worth looking into a real portable recovery tool instead.

Your second option is to use the installer and do an "import configuration", which will recover all your configs, backups, ssh keys, dhcpleases and captive portal databases as long as the disk isn't corrupted badly. After import, "quick and easy" install will install the system cleanly with all your settings in place.

If anyone has other suggestions or improvements I'd like to bring them to the table to find a sensible solution.


Cheers,
Franco

PS: https://github.com/opnsense/tools/issues/10

[gratis]

Hello franco,

The sync option should work for my use case, as I am currently using a re-purposed ThinkCentre desktop with a standard HDD as a test machine. Thanks for the tip.

Only a week into testing, and didn't have a large amount time invested in configuring OPNsense, so I simply reinstalled and started over fresh.

However, your suggestions on how to recover look promising. In retrospect I should have switched to a backup router and attempted to resolve the issue with your help, to test your suggestions. Next time...

Thanks again for your efforts.

[cmb]

The root cause is an issue with pw not issuing fsync where it should, leaving the files in an inconsistent state, where if you have an unclean shut down within some time after touching the passwd or group file, you'll end up with them being blank, or if not using SU, with random bits of other files in /etc/.

Same happens on stock FreeBSD if it occurs shortly after running pw. One of our (pfSense) developers will get a fix for pw into FreeBSD, but for the time being, setting sync is confirmed to 100% resolve the problem. That's a reasonable solution for firewall use cases.

It's easily replicable on OPNsense and pfSense because we write the passwd and group files on every boot, which is much more than what a stock FreeBSD install would generally do. Nature of the beast for what we're doing, though we'll make it idempotent in the future in pfSense.

Since setting sync, we have systems that have been through several thousand power cycles (snmpset to IP PDU scripted in a loop for days and days) immediately after writing the passwd and group files (which is what triggers the problem), and have had 0 problems.

There is a much worse problem of some sort with OPNsense that I hit the very first time I pulled the power plug on it. Clean install, boot to the console menu, yank the power plug. Completely trashed filesystem. Couldn't touch it without kernel panic (attached). The only relevant diff I saw between OPNsense, stock FreeBSD, and pfSense in that regard is you're running SU without J. FreeBSD 10.1 is SU+J by default, pfSense 2.2.3 and newer is SU+J by default (and pre-2.2.3 was no SU, no J). I went through several hundred of power cycles without sync in tracking the root cause, only a handful of those on OPNsense as another point of comparison, but it's the only thing that ever ended up with anything worse than blank or corrupted passwd and group files. Guessing you probably want SU+J so you're out of what's probably a much less tested code path, at least if my suspicion of it being SU without J is down the right path.

[jschellevis]

Hi Chris,

Thanks for your help and explanation.
The cause is clear now.

Any idea when your developer will have a patch?
Do you guys need any help with getting it ready for upstream?

[franco]

gratis, apcupsd has been added and will be available with 15.7 as an optional package: https://github.com/opnsense/tools/commit/e82e6356d1aa6960413699d9bff1fe957c8cffc1

[cmb]

JimT is testing and reviewing his fix as time permits. A bit more work on that and we'll put it into our power cycle test harness and get it upstream once confirmed, hopefully MFCed into FreeBSD 10.2. We'll have no issue getting it in upstream once we're confident it's correct, multiple FreeBSD committers already involved.

[jschellevis]

Thanks Chris,

Franco has created a workaround that I am currently testing.
https://github.com/opnsense/core/commit/1b7aec7a7738a99eaa567b00adfc8b9c983dd86b

[franco]

Chris, thanks for your help here. Unfortunately, forcing everyone to sync is suboptimal for a bug that happens in just one utility. I agree that pw must be fixed mid-term. Now, given that Jim has openly mocked my commit that Jos confirmed actually working I'm not going to take anything but the real pw fix.

I feel that said negativity will keep preventing us from working together in the future.

Thanks for your understanding. Looking forward to your fix in 10-STABLE.

[gratis]

Looks like sharing the backup story started an interesting exchange... It's good that Chris pointed out the root of the problem, and that there are multiple solutions in the works.

Thanks for the addition franco, hopefully others will benefit from it aswell...

[franco]

We've discussed this a couple more times internally and have come to the conclusion that this issue is not fixable, or at least not in the way it has been presented and discussed. While it's true that "sync" completely circumvents the issue, it seems that UFS has gotten a lot more error prone in FreeBSD 10 because of a yet to be discovered regression. We do not intend to switch our installs to "sync" or use journaling on top of soft updates.

FreeBSD's default for mounting is "noasync" which is a hybrid approach for writing data to disk: metadata is synchronously written, while file and directory data is held in RAM for a while, presenting the writer with a finished write even though the data is still pending a flush to the physical disk. While looking at pw and the supposed fsync() missing (fflush() really, because it doesn't use the open() function call), we've noticed that pw itself uses libc by default, so "fixing pw" does not look like a sensible route. What makes this worse now is that all other utilities using libc with the particular user/group function have this potential corruption issue too. And then, still, while digging deeper, we've found that other files do indeed corrupt as well, especially when pulling the plug on a booting system (e.g. /etc/shells), eventually rendering the installations impossible to be accessed even from a serial console.

So now, we've put in place a mechanism that ensures the system recovers the *important* files on boot so that a login capability is ensured and a fully operational system can also recover, but -- keeping the above it mind -- any unclean shutdown may lead to a file corruption at some point most likely rendering the system unable to recover or start due to missing configuration files or binaries. We've also made sure that the syncer kernel thread has lower timeouts (we went from 30 seconds previously to 5 seconds) for file and directory writes. The latter is a moving target as we monitor and tweak performance.

In all that we do, we try to avoid adding kernel/base patches or reboots or changing the behaviour away from FreeBSD defaults. If that's a problem, please speak up now. Otherwise, enjoy the upcoming 15.7.

Our patches for reference:

https://github.com/opnsense/core/commit/81edf54f2afb643800becd8ce231b4d891d97c77
https://github.com/opnsense/core/commit/cc7180222db0dd67fb4857fb43fa28cf4b5c27e9
https://github.com/opnsense/core/commit/7baccb7c42dd812963a931edb2483a25269eaf7a
https://github.com/opnsense/core/commit/f45171e7851b26acbd23b48c8c30d3f35b997e71
https://github.com/opnsense/core/commit/50613a493692ca6f43eb31e52403b1f97523ecb6

[chol]

  Like that:

In all that we do, we try to avoid adding kernel/base patches or reboots or changing the behaviour away from FreeBSD defaults.

OPNsense 15.7 .. steady as she goes!

[chol]

Re: UFS SU+J sync

Data loss due to vfs buffering


FreeBSD 10.1 - Deadlock on reboot with UFS tuned with SU+J


Hang on shutdown/root unmount after FreeBSD 10.1R


Features and status of FreeBSD's Ext2 implementation - BSDCan 2014

[franco]

Features and status of FreeBSD's Ext2 implementation - BSDCan 2014

Ad and me are happy about this in a way you can't imagine.

[franco]

To be fair, here is what pfSense has to say. https://blog.pfsense.org/?p=1815

The timing is impeccable; if I didn't know any better it would look like they did not want 15.7 to have the fix. A day earlier and we would have picked it up gladly.