OPNsense Forum
English Forums => Zenarmor (Sensei) => Topic started by: rfc805 on May 16, 2023, 08:25:19 pm
-
I don't have too much information to really provide on this. However, when performing the upgrade to 23.1.7 today, it reached upgrading Zenarmor/os-sensei to 1.13. At this point it gave a message that it was saving state as it was running, and then "Waiting for PIDs: ..." - at this point the opnsense system went entirely unreachable and stopped forwarding traffic entirely. After waiting for several minutes in this state, I had to use out of band console to go in and kill all eastpect processes, at which point the OPNsense system functioned again.
-
True, I had this several times: installing Zenarmor kills LAN connections. I had to use the console to install it properly (just in case, I have reinstalled it).
-
I am running into a similar issue. Out of curiosity, what type of interface/nic are you using?
-
Various:
- a computer with Intel Corporation 82576 Gigabit Network Connection( rev 01) and Intel Corporation Ethernet Connection I217-V (rev 05) and Qualcomm Atheros AR9462 Wireless Network Adapter (rev 01)
- a Proxmox with Intel Corporation 82576 Gigabit Network Connection (rev 01) available through PCI devices (no qemu plugin installed)
- APU2 with Intel(R) I210 Flashless (Copper)
So, it happened everytime I had to update Zenarmor to the present version.
-
Hi All,
What is the output of /tmp/zenarmor_update.log? Do you use Suricata as well?
-
I do use Suricata as well, but obviously on different interfaces.
There is no such log file on my host, only a zenarmor_updates.json
-
Quick question: were you upgrading from 23.1.6 or from an earlier OPNsense version?
This seems netmap-related and I wonder if you were upgrading from an earlier version of OPNsense where the netmap emulated fixes were not there...
-
No, I don't use Suricata.
-
I was upgrading from 23.1.6
-
I have à similar issue. Don’t upgrade. This is Bullshit.
Everything down.
I don’t use suricat.
I have a protection i7
-
23-7-1
-
@rfc805, @almodovaris, @TritonB7
Can you reach out to Zenarmor support? Use the "Bug Report" screen on the upper right hand corner of the screen - in case you're not familiar.
Make sure you send all OPN/Zenarmor config and logs.
This looks pretty much like another netmap thing. We want to investigate further.
Thanks.
-
Sorry, I have meanwhile uninstalled Zenarmor, deleted all its folders, and then reinstalled it (on all machines). So, I guess logs no longer contain that information. Just that you know, on Proxmox I use now emulated netmap driver.
-
@rfc805, @almodovaris, @TritonB7
Can you reach out to Zenarmor support? Use the "Bug Report" screen on the upper right hand corner of the screen - in case you're not familiar.
Make sure you send all OPN/Zenarmor config and logs.
This looks pretty much like another netmap thing. We want to investigate further.
Thanks.
I did submit a Bug Report earlier this month along with config and logs. I'm using an Intel x550-t2 and i350t4v2, same issue with both cards.
-
update:
i receive an answer from helpdesk:
Thanks for reaching out and letting us know about the problem.
"
Is the FW reachable? Can you try to stop Zenarmor packet Engine by runnig the command "service eastpect stop" on the console as root? Then please change the deployment mode to emulated driver in Configuration - General - Deployment Mode - L3 Routed mode with netmap emulated driver on GUI.
"
I start a new post because i was unable to reinstall zenarmor
-
Submitted a bug, got a blow off "we can't reproduce, go away" response. Meh - think it's best to just avoid using Zenarmor in the future.
-
Hi,
Can you share the ticket ID? It could be a misunderstanding. We try to solve the issue via remote session if we can not reproduce it.
-
How would you solve the issue via a remote session for an upgrade failure? Doesn't make much sense in the approach.
I understand it as a general policy to troubleshooting, but can't see how it's relevant for this one.
It's also a bit of an uncomfortable approach with a firewall device.
-
I cannot get crash dumps, but every time I reinstall Zenarmor I get a kernel panic after Application category migration... done.
Panic happens each and every time I install Zenarmor, even if I had cleaned its every trace before reinstalling it.
-
Hi Almodovaris,
Could you share the kernel panic message and the zenarmor* files under /tmp/ folder with the Zenarmor team?
Bests
-
Had this as well, intel nics. This has happened on these nics for over a year, I did reach out to support who did offer to remote in, which is hard to do when interfaces are all down ;) I uninstalled, but have you tried emulated mode? Ive heard emulated is pretty stable now...
-
This happens before eastpect is even started.
I get a kernel panic and OPNsense reboots. There is nothing about kernel panic in /tmp. /tmp is volatile. If you show me how to capture Proxmox console, I can share the output, there are many pages of errors.
-
Okay, made /tmp and /var stick. Still no logs in /usr/local/sensei/support/crash_dumps or /var/crash.
Nothing interesting in /tmp either. Nor in /var/log .
-
https://youtu.be/dRZDPivYrDA
At 01:30 begins the action.
-
Hi,
It is crashing when tries to stop the engine. Can you try to remove /usr/local/sensei folder and kill all easpect process?
-
Already did that, several times. I had reinstalled Zenarmor after deleting its every folder and rebooting OPNsense before installing it. I still had the same crash.
-
Hi,
Is there a chance to share screen video during the fresh install. In the previous video it was crashing while tries to stop Zenarmor engine. I wonder what is the cause of crash in fresh install.
-
Done.
https://youtu.be/QB4WGX-qnh8
Action begins at 01:40.
-
Hi @almodovaris,
Thanks for the video. It was very helpful in understanding what's going on.
It looks like the crash is indeed caused by the 'sysctl' binary while trying to update net.inet.ip.intr_direct_queue_maxlen sysctl value.
Although the crash is not directly caused by zenarmor per se; something along the package install path seems to be triggering an operating system bug leading to an OS panic.
We're trying to reproduce this on our environment.
-
Well, it's not hard to reproduce on an APU2E4.
-
Now it's even worse: simply accessing https://192.168.1.1/system_advanced_sysctl.php crashes OPNsense (you do have to enter username and password if asked).
If it matters, I have OPNsense 23.1.9_19-amd64.
kernel and base are 23.1.8_5.
Oddly enough, getting rid of maxlen from config.xml does not solve the problem.
Even
# sysctl -a
crashes the system.
Reverting to stable 22.1.8 base and kernel, sysctl -a no longer crashes my system. And reinstalling Zenarmor no longer crashes my system. Please note that the crash happened also on stable versions, I think that removing maxlen from config.xml solved the problem for 23.1.8 base and kernel.
For the record, that tunable was set as default.
-
I've checked:
opnsense-update -zbr 23.1.8_5 is safe;
opnsense-update -zkr dbg-23.1.8_5 isn't safe.
-
The last stable version base/kernel is 23.1.8. If you install random snapshots (opnsense-update -z) I'm unsure what you are trying to achieve...
If this is a FreeBSD 13 issue it's probably going to happen all the time since we don't alter code that is relevant here.
Cheers,
Franco
-
Let's put it this way: I have an addiction, I'm addicted to updates.
-
Cowbell vibes, but I digress...
Ok, fair enough. You might want to try FreeBSD 13.2?
# opnsense-update -zbkr 23.7.d
It's unlikely an upstream fix will automatically find its way to 13.1/23.1 nowadays.
Cheers,
Franco
-
23.7.d works okay.
-
Ok, that's nice to hear.
Cheers,
Franco