[CALL FOR TESTING] Netmap generic mode queue stall fixes

Started by franco, January 27, 2023, 11:38:45 AM

Previous topic - Next topic
Now it gives me stable/23.1-n250429-c163ff33fa8.
OPNsense HW:

Minisforum Venus series UN100C, 16 GB RAM, 512 GB SSD
T-bao N9N Pro, 16 GB RAM, 512 GB SSD

That would be 23.1.5-netmap2 I uploaded this morning for internal testing ;)

But that is the final state on the stable branch so better to test by anyone who can:

# opnsense-update -zkr 23.1.5-netmap2 && opnsense-shell reboot


Cheers,
Franco

Installed, working for now. I'll have to wait 2-3 days to see if it drops again or not.
OPNsense 24.7.7  - QEMU/KVM (Ubuntu), i9-9900K 16 core @ 5ghz, 16GB RAM, 64GB SSD, 2 dedicated SFP+ NICs

I'll have to restart later today but all my vlan interfaces are having problems with this update using both emulated and native in Zenarmor with ipv4, ipv6 seems to be fine.

Restarting individual services didn't help, only stopping zenarmor did. So I will just restart the whole router later tonight when I can

So Vlans are working now, but I've had 2 complete loss of internet today since being on Zenarmor emulated with the new 23.1.5-netmap2. I wasn't home so I couldn't tell if it was ZenArmor or Suricata that was the issue.
Before the netmap upgrade in the past I would only lose internet inside and I could use Zenarmors cloud portal to restart it on my router and bring internet back up, or wireguard and get into my router to restart it. Now with the new netmap I'm losing control remotely, I can't use wireguard or Zenarmor's page to restart it.

I'm up four days now after applying the patch (correctly, don't think I did the first time) and using netmap emulator for ZenArmor config. This is the longest I've been running since upgrading to 23.1. Things have been, dare I say it, stable? Fingers crossed. :)
OPNsense 24.7.7  - QEMU/KVM (Ubuntu), i9-9900K 16 core @ 5ghz, 16GB RAM, 64GB SSD, 2 dedicated SFP+ NICs

We are looking for internal approval between participating parties on the last published state for 23.1.6. Overall it looks like we are better off with the patches than without and we likely won't get broader feedback otherwise. If not I expect 23.1.7 to have it in a few weeks.


Cheers,
Franco

Thanks Franco.

Hate to report though that with all of the right things in place, I still dropped early this morning and had to restart ZenArmor to resolve it. It was a longer uptime duration this time, but it still wound up dropping packets on LAN. :/
OPNsense 24.7.7  - QEMU/KVM (Ubuntu), i9-9900K 16 core @ 5ghz, 16GB RAM, 64GB SSD, 2 dedicated SFP+ NICs

Not much to be sorry about. The current state of the project is better than what we had before so either we ditch it or move forward. We are going to do the latter. ;)


Cheers,
Franco

I've unstickied this and removed the remaining test kernels since 23.1.6 has all of it. Feel free to respond here with enough information to continue discussion (log files, setup, previous experience).


Cheers,
Franco

April 23, 2023, 04:29:39 PM #85 Last Edit: April 24, 2023, 03:55:21 PM by dfw3xam1n3r
[EDIT: Franco: Just realized you said it will be in 23.1.7, not .6. Nevermind!]

My connection still dropped. I don't know why. I was away when it happened and was able to bring it back up remotely. Restarting ZenArmor didn't help though, only a reboot. Currently have ZenArmor set to monitor only for now.

Just for my confirmation, I've upgraded to 23.1.6, and re: ZenArmor (when I take it out of monitoring only) I'm supposed to be using the emulated netmap driver not the native correct? Do I need to do anything with IDS/IPS/Suricata since I'm running that as well?
OPNsense 24.7.7  - QEMU/KVM (Ubuntu), i9-9900K 16 core @ 5ghz, 16GB RAM, 64GB SSD, 2 dedicated SFP+ NICs

Hi @dfw3xam1n3r
Did you test Zenarmor with Routed (L3 Mode, Reporting and Blocking available) with emulated netmap driver on OPNsense 23.1.6 and have any issues? Some users reported that their problems are resolved with this configuration.

I tested with emulated and within a couple/few days the same thing happens. Interestingly the failure mode was the same but usually a restart of all services (using native) brings everything back up. When doing emulated, one of my vlan's didn't come back and I had errors in the console of the emulated netmap adapter interface being unavailable. A reboot brought everything back up.

Quote from: beki on April 27, 2023, 09:48:42 AM
Hi @dfw3xam1n3r
Did you test Zenarmor with Routed (L3 Mode, Reporting and Blocking available) with emulated netmap driver on OPNsense 23.1.6 and have any issues? Some users reported that their problems are resolved with this configuration.

Yeah I did and the same thing happened, so I'm just in monitoring mode until 23.1.7 comes out.
OPNsense 24.7.7  - QEMU/KVM (Ubuntu), i9-9900K 16 core @ 5ghz, 16GB RAM, 64GB SSD, 2 dedicated SFP+ NICs

As said elsewhere whatever you expect for 23.1.7 is not in 23.1.7 because "it" does not exist.


Cheers,
Franco