PHP Fatal error: Uncaught OPNsense\Core\ConfigException: empty file

[bacanol]

Hello everyone,

since the last update to Business Edition latest, I get this error when accessing web-interface and when trying to access via ssh:

Code: [Select]

PHP Fatal error:  Uncaught OPNsense\Core\ConfigException: empty file in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:405
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(436): OPNsense\Core\Config->loadFromStream(Resource id #18)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(646): OPNsense\Core\Config->load()
#2 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(354): OPNsense\Core\Config->restoreBackup('/usr/local/etc/...')
#3 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Singleton.php(51): OPNsense\Core\Config->init()
#4 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Singleton.php(72): OPNsense\Core\Singleton->__construct()
#5 /usr/local/etc/inc/config.inc(121): OPNsense\Core\Singleton::getInstance()
#6 /usr/local/etc/inc/config.inc(364): parse_config()
#7 /usr/local/opnsense/scripts/shell/banner.php(32): require_once('/usr/local/etc/...')
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 405

Fatal error: Uncaught OPNsense\Core\ConfigException: empty file in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php:405
Stack trace:
#0 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(436): OPNsense\Core\Config->loadFromStream(Resource id #18)
#1 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(646): OPNsense\Core\Config->load()
#2 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php(354): OPNsense\Core\Config->restoreBackup('/usr/local/etc/...')
#3 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Singleton.php(51): OPNsense\Core\Config->init()
#4 /usr/local/opnsense/mvc/app/library/OPNsense/Core/Singleton.php(72): OPNsense\Core\Singleton->__construct()
#5 /usr/local/etc/inc/config.inc(121): OPNsense\Core\Singleton::getInstance()
#6 /usr/local/etc/inc/config.inc(364): parse_config()
#7 /usr/local/opnsense/scripts/shell/banner.php(32): require_once('/usr/local/etc/...')
#8 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Core/Config.php on line 405

Do you have any idea, how to get back to regular operation?

[bacanol]

addition: the serial console is broken also. I am wondering what php needs to be there?

[franco]

Sounds like the file system disintegrating. If you can't log in that's a big problem in terms of fixing it easily.

I think it needs a reinstall. Rebooting it will probably make it stop working, too.


Cheers,
Franco

[doktornotor]

Reinstall would be also a good opportunitely to not use UFS. Assuming it's filesystem issue and not failing HW.

[franco]

It could be UFS coupled with a power outage, but it's not entirely clear although "empty file" points to the usual UFS issue on power outages indeed.

The preinstalled hardware comes with business version on ZFS nowadays.


Cheers,
Franco

[bacanol]

Hello and thank you for your answers up until now.

It is a Deciso-Box with business edition on it. It seems a bit to me as if the file system space is limited somehow after the update. Is there any rescue procedure that I might follow to examine the file system with a USB boot medium for example?

[franco]

Yeah, you can get an image and use live mode with the config importer to run a fsck on the disk (it does that automatically when importing). But chances of recovery are slim so an in-place reinstall after config import likely makes the most sense. Make sure to grab a business image from https://opnsense-update.deciso.com/ so you don't have to go through the community image.

Not knowing how old that box is still leaves us guessing at UFS/ZFS.


Cheers,
Franco

[bacanol]

Hello again,

I managed to clean up space (surricata was filling up all space) and get the device back on line. Btw it is a decent DEC3860 v2 with zfs. It still had the php errors on the web access, but I managed to get a backup config into the right place. I ran the audits and nothing suspect was brought to my attention:

Code: [Select]

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 23.10_2 at Wed Nov  8 12:21:18 CET 2023
>>> Check installed kernel version
Version 23.7.4 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 23.7.4 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense
>>> Check installed plugins
os-OPNBEcore 1.2
os-acme-client 3.19
os-apcupsd 1.1
os-ddclient 1.16
os-etpro-telemetry 1.6_1
os-git-backup 1.0_3
os-haproxy 4.1_1
os-intrusion-detection-content-et-open 1.0.1
os-intrusion-detection-content-snort-vrt 1.1_1
os-net-snmp 1.5_2
os-wireguard 2.3
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense-business" has 69 dependencies to check.
Checking packages: ...................................................................... done
***DONE***

I guess everything is fine again. I will now fine-tune surricata to prevent this from happening again. Thank you for your support.

[franco]

Ok nice to hear. Lucky it was only the disk filling up then. Crisis averted.


Cheers,
Franco

[bacanol]

Just for the curious readers, I limited the dataset size with:

Code: [Select]

zfs set quota=300G zroot/var/log
So the log partition will not fill up the full 450G storage. ;-) Thank you for all the help!