PHP Fatal error: Uncaught OPNsense\Core\ConfigException: empty file

[bacanol]

Hello everyone,

since the last update to Business Edition latest, I get this error when accessing web-interface and when trying to access via ssh:

Code Select Expand


PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: empty file in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:405
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(436): OPNsense\Core\Config->loadFromStream(Resource id #18)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(646): OPNsense\Core\Config->load()
#2 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(354): OPNsense\Core\Config->restoreBackup('/usr/local/etc/...')
#3 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Singleton.php(51): OPNsense\Core\Config->init()
#4 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Singleton.php(72): OPNsense\Core\Singleton->__construct()
#5 /usr/local/etc/inc/config.inc(121): OPNsense\Core\Singleton::getInstance()
#6 /usr/local/etc/inc/config.inc(364): parse_config()
#7 /usr/local/opnsense/scripts/shell/banner.php(32): require_once('/usr/local/etc/...')
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 405

Fatal error: Uncaught OPNsense\Core\ConfigException: empty file in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:405
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(436): OPNsense\Core\Config->loadFromStream(Resource id #18)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(646): OPNsense\Core\Config->load()
#2 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(354): OPNsense\Core\Config->restoreBackup('/usr/local/etc/...')
#3 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Singleton.php(51): OPNsense\Core\Config->init()
#4 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Singleton.php(72): OPNsense\Core\Singleton->__construct()
#5 /usr/local/etc/inc/config.inc(121): OPNsense\Core\Singleton::getInstance()
#6 /usr/local/etc/inc/config.inc(364): parse_config()
#7 /usr/local/opnsense/scripts/shell/banner.php(32): require_once('/usr/local/etc/...')
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 405


Do you have any idea, how to get back to regular operation?

[bacanol]

addition: the serial console is broken also. I am wondering what php needs to be there?

[franco]

Sounds like the file system disintegrating. If you can't log in that's a big problem in terms of fixing it easily.

I think it needs a reinstall. Rebooting it will probably make it stop working, too.


Cheers,
Franco

[doktornotor]

Reinstall would be also a good opportunitely to not use UFS. Assuming it's filesystem issue and not failing HW.

[franco]

It could be UFS coupled with a power outage, but it's not entirely clear although "empty file" points to the usual UFS issue on power outages indeed. ;)

The preinstalled hardware comes with business version on ZFS nowadays.


Cheers,
Franco

[bacanol]

Hello and thank you for your answers up until now.

It is a Deciso-Box with business edition on it. It seems a bit to me as if the file system space is limited somehow after the update. Is there any rescue procedure that I might follow to examine the file system with a USB boot medium for example?

[franco]

Yeah, you can get an image and use live mode with the config importer to run a fsck on the disk (it does that automatically when importing). But chances of recovery are slim so an in-place reinstall after config import likely makes the most sense. Make sure to grab a business image from https://opnsense-update.deciso.com/ so you don't have to go through the community image.

Not knowing how old that box is still leaves us guessing at UFS/ZFS.


Cheers,
Franco

[bacanol]

Hello again,

I managed to clean up space (surricata was filling up all space) and get the device back on line. Btw it is a decent DEC3860 v2 with zfs. It still had the php errors on the web access, but I managed to get a backup config into the right place. I ran the audits and nothing suspect was brought to my attention:

Code Select Expand


***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 23.10_2 at Wed Nov  8 12:21:18 CET 2023
>>> Check installed kernel version
Version 23.7.4 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 23.7.4 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense
>>> Check installed plugins
os-OPNBEcore 1.2
os-acme-client 3.19
os-apcupsd 1.1
os-ddclient 1.16
os-etpro-telemetry 1.6_1
os-git-backup 1.0_3
os-haproxy 4.1_1
os-intrusion-detection-content-et-open 1.0.1
os-intrusion-detection-content-snort-vrt 1.1_1
os-net-snmp 1.5_2
os-wireguard 2.3
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense-business" has 69 dependencies to check.
Checking packages: ...................................................................... done
***DONE***


I guess everything is fine again. I will now fine-tune surricata to prevent this from happening again. Thank you for your support.

[franco]

Ok nice to hear. Lucky it was only the disk filling up then. Crisis averted. :)


Cheers,
Franco

[bacanol]

Just for the curious readers, I limited the dataset size with:

Code Select Expand

zfs set quota=300G zroot/var/log

So the log partition will not fill up the full 450G storage. ;-) Thank you for all the help!