OPNSense throwing multiple PHP errors during update

[BigFreddy]

Hi,

My firewall not long ago started throwing minor erros from time to time when I have been updating it and it usually gave me the following red message on the screen: "Danger, unexpected error, check log for details." followed by the successful update with no errors even after I refreshed the page during the update. However, this time it got worse and I'm starting to worry about my firewall and its state to the point where I'm wondering if I should just re-install it alltogether, on the surface everything works fine as I checked a few things but still what happened is seriously worrying me.

When I started the update process this time around, I got the same red error message as before: "Danger, unexpected error, check log for details." with the following action still taking place for a very long time:

[19/93] Extracting liblz4-1.10.0_2,1:.

As the update was stuck on the above action taking place and the progress dot not moving at all for a long time, I then decided to refresh the web page in the browser, which at first gave me 403 Access Forbidden followed by 404 Not Found errors when I was refreshing the web page like a mad man to check if the update killed my firewall entirely. The mentioned error codes worried me that my internal devices got exposed to the internet due to the failure, also I'm not quite sure how OPNsense handles failures when something malfunctions. Once the firewall became online as I presume it rebooted, I checked a few things and everything runs fine so I don't see any issues.

During the time when the above issues with errors took place, I managed to note down the time frame of it taking place and upon checking logs on the firewall, I got the following errors, many of which were repeated in the logs at least a few times:

General Logs:

Code Select Expand


Notice kernel <6>[9338999] pid 62018 (sh), jid 0, uid 0: exited on signal 10 (no core dump - bad address)
Error opnsense /usr/local/sbin/pluginctl: The command `/sbin/ifconfig -Lmv ` failed to execute ()
Error php #0 /usr/local/opnsense/scripts/health/library/OPNsense/RRD/Factory.php(123): ReflectionClass - >_construct('\\OPNsense\\RRD\\T...')
Error php Stack trace:
Error php Error instatiating Traffic [ReflectionException: Class "\OPNsense\RRD\Types\Traffic" does not exist in /usr/local/opnsense/scripts/health/library/OPNsense/RRD/Factory.php:123
Error php #2 {main}]
Error php #1 /usr/local/opnsense/scripts/health/updaterrd.php(64): OPNsense\RRD\Factory->updateAll(false)

Backend Logs:

Code Select Expand

Error configd.py [redacted-id] Script action stderr returned "b'Bus error'"
Error configd.py [redacted-id] Script action failed with Command '/usr/local/opnsense/scripts/routes/gateway_status.php' died with <Signals.SIGBUS: 10>. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 89, in execute subprocess.run(script_command, env=self.config_environment,shell=True, File "/usr/local/lib/python3.11/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args,subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/routes/gateway_status.php' died with <Signals.SIGBUS: 10>.

Is there anything in the above errors that I should worry about ? I don't really understand why my firewall is throwing all the errors, my setup is fairly basic and vanilla with no additional applications or modifications outside of what the web interface offers.

[Patrick M. Hausen]

This screams "hardware problem". Either memory, CPU, overheating, ... or a dying SSD/disk.

What does "dmesg" output?

[BigFreddy]

This screams "hardware problem". Either memory, CPU, overheating, ... or a dying SSD/disk.

What does "dmesg" output?

What makes it obvious it's a hardware problem ? Is it the "b'Bus error'" error ?
How can I check dmesg output ?

[Patrick M. Hausen]

You type D M E S G followed by the ENTER key after logging in to the firewall via SSH or connected to the console and selecting "8" for a shell. All lower case letters.

[BigFreddy]

You type D M E S G followed by the ENTER key after logging in to the firewall via SSH or connected to the console and selecting "8" for a shell. All lower case letters.

Is there a way to do this from the web interface ?

[Patrick M. Hausen]

No. Use the web interface to enable SSH and log in. There is SSH for every desktop operating system imaginable.

[BigFreddy]

You type D M E S G followed by the ENTER key after logging in to the firewall via SSH or connected to the console and selecting "8" for a shell. All lower case letters.

I tried doing that and it only shows me the logs after the reboot took place I think.

Here are the errors I get when I view dmesg:

Code Select Expand

WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0.

acpi0: Power Button (fixed)
Unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0

atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O

ns8250: UART FCR is broken (This one appears 3 times)

ahcicho 0: Timeout on slot 28 port 0
CAM Status: Command Timeout
Retrying command, 3 more tries remain

ahcicho 0: Timeout on slot 7 port 0
CAM Status: Command Timeout
Retrying command, 2 more tries remain

ahcicho 0: Timeout on slot 12 port 0
CAM Status: Command Timeout
Retrying command, 1 more tries remain

ahcicho 0: Timeout on slot 17 port 0
CAM Status: Command Timeout
Retrying command, 0 more tries remain

ahcicho 0: Timeout on slot 22 port 0
CAM Status: Command Timeout
Error 5, Retries exhausted


[Patrick M. Hausen]

Your SSD/disk is dying. Save a configuration while you still can, replace the drive, reinstall, restore configuration.

[BigFreddy]

Your SSD/disk is dying. Save a configuration while you still can, replace the drive, reinstall, restore configuration.

Which part makes it obvious ? It's interesting because I always got the following errors (ahcicho 0: Timeout on slot 28 port 0, CAM Status: Command Timeout , Retrying command, 3 more tries remain) since I got the box and what it only caused was slower reboot time trying to do TRIMs I think which some SSDs don't support so I presumed my SSD didn't support that feature and only recently started causing issues, hopefully it's my SSD and nothing else.

Do you know if the vanilla configuration file is backwards compatible with older versions of OPNsense ?

[Patrick M. Hausen]

I have only ever seen these:

Code Select Expand

ahcicho 0: Timeout on slot 7 port 0
CAM Status: Command Timeout
Retrying command, 2 more tries remain
with dying devices. If I saw that in a new unit I would never put that into production before I had successfully eliminated the cause. Timeouts in the CAM subsystem must not happen. If they do, something is broken. Never ignore them.

What do you mean by "vanilla" and "backwards compatible"? Save the configuration from your current unit, fix the hardware, install the very same version, restore configuration ...

[franco]

> Notice kernel <6>[9338999] pid 62018 (sh), jid 0, uid 0: exited on signal 10 (no core dump - bad address)
> Error opnsense /usr/local/sbin/pluginctl: The command `/sbin/ifconfig -Lmv ` failed to execute ()
> Error configd.py [redacted-id] Script action stderr returned "b'Bus error'"

I agree with Patrick. This is looks very unlike a healthy system.

The firmware health audit can probably confirm?


Cheers,
Franco