OPNsense upgrade failure with Zenarmor (23.1.7)

[rfc805]

Submitted a bug, got a blow off "we can't reproduce, go away" response.  Meh - think it's best to just avoid using Zenarmor in the future.

[sy]

Hi,

Can you share the ticket ID? It could be a misunderstanding. We try to solve the issue via remote session if we can not reproduce it.

[rfc805]

How would you solve the issue via a remote session for an upgrade failure?  Doesn't make much sense in the approach. 

I understand it as a general policy to troubleshooting, but can't see how it's relevant for this one.

It's also a bit of an uncomfortable approach with a firewall device.

[almodovaris]

I cannot get crash dumps, but every time I reinstall Zenarmor I get a kernel panic after Application category migration... done.

Panic happens each and every time I install Zenarmor, even if I had cleaned its every trace before reinstalling it.

[beki]

Hi Almodovaris,

Could you share the kernel panic message and the zenarmor* files under /tmp/ folder with the Zenarmor team?

Bests

[jclendineng]

Had this as well, intel nics.  This has happened on these nics for over a year, I did reach out to support who did offer to remote in, which is hard to do when interfaces are all down I uninstalled, but have you tried emulated mode? Ive heard emulated is pretty stable now...

[almodovaris]

This happens before eastpect is even started.

I get a kernel panic and OPNsense reboots. There is nothing about kernel panic in /tmp. /tmp is volatile. If you show me how to capture Proxmox console, I can share the output, there are many pages of errors.

[almodovaris]

Okay, made /tmp and /var stick. Still no logs in /usr/local/sensei/support/crash_dumps or /var/crash.

Nothing interesting in /tmp either. Nor in /var/log .

[almodovaris]

https://youtu.be/dRZDPivYrDA

At 01:30 begins the action.

[sy]

Hi,

It is crashing when tries to stop the engine. Can you try to remove /usr/local/sensei folder and kill all easpect process?

[almodovaris]

Already did that, several times. I had reinstalled Zenarmor after deleting its every folder and rebooting OPNsense before installing it. I still had the same crash.

[sy]

Hi,

Is there a chance to share screen video during the fresh install. In the previous video it was crashing while tries to stop Zenarmor engine. I wonder what is the cause of crash in fresh install.

[almodovaris]

Done.

https://youtu.be/QB4WGX-qnh8

Action begins at 01:40.

[mb]

Hi @almodovaris,

Thanks for the video. It was very helpful in understanding what's going on.

It looks like the crash is indeed caused by the 'sysctl' binary while trying to update net.inet.ip.intr_direct_queue_maxlen sysctl value.

Although the crash is not directly caused by zenarmor per se; something along the package install path seems to be triggering an operating system bug leading to an OS panic.

We're trying to reproduce this on our environment.

[almodovaris]

Well, it's not hard to reproduce on an APU2E4.