OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: mb on September 16, 2020, 06:53:51 pm

Title: Call for testing: official netmap kernel
Post by: mb on September 16, 2020, 06:53:51 pm
Dear OPNsense community,

It's my pleasure to announce that OPNsense team has shipped the official netmap test kernel today.

This kernel fixes important stability and reliability issues with regard to vmx(4), vtnet(4), ixl(4), ix(4) and em(4) ethernet drivers.

The  kernel also adds long-awaited support for tun(4) and lagg(4) interfaces.

The end benefit of this kernel is that you'll be able to run Sensei or Suricata on the following:

To deploy the new kernel just run below command and restart your firewall.

Code: [Select]
# opnsense-update -kr 20.7.3-netmap
Patches which went into this kernel have been under heavy testing by us (Sunny Valley Networks) and by the OPNsense team for a few weeks now.

We'd very much appreciate your further testing and feedback. If no further issues pop up, OPNsense team will be shipping all these functionality with 20.7.4 or later releases.

As Sunny Valley Networks, we'd very much like to thank OPNsense/HardenedBSD team, netmap team (Universita di Pisa) and the FreeBSD team for their awesome collaboration and precious efforts. With their full coordination and co-operation, we are able to provide this today.
Title: Re: Call for testing: official netmap kernel
Post by: mimugmail on September 16, 2020, 09:44:01 pm
Good news! Today in Webinar I was asked about vmx driver status. Thanks for your efforts!! :)
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 16, 2020, 09:46:04 pm
All welcome @mimugmail. Enjoy ;)
Title: Re: Call for testing: official netmap kernel
Post by: andrema2 on September 16, 2020, 09:59:27 pm
# opnsense-update -kr 20.7.2-netmap

Just one question, maybe a silly one, is this reversable in case of problems ?

Thanks
Title: Re: Call for testing: official netmap kernel
Post by: binaryanomaly on September 16, 2020, 10:34:26 pm
Awesome!
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 16, 2020, 10:35:40 pm
Just one question, maybe a silly one, is this reversable in case of problems ?

No, actually a good question. Yes, you can:

Code: [Select]
# opnsense-update -kr 20.7.2
Title: Re: Call for testing: official netmap kernel
Post by: GreenMatter on September 16, 2020, 11:03:36 pm
# opnsense-update -kr 20.7.2-netmap
Just one question, maybe a silly one, is this reversable in case of problems ?
And can I apply this update directly on 20.1.9 or it's better to wait for 20.7.3?
Title: Re: Call for testing: official netmap kernel
Post by: binaryanomaly on September 16, 2020, 11:04:07 pm
@mb

Is it to be expected that now with "20.7.2-netmap" the update functionality proposes to update to "20.7.2"

Title: Re: Call for testing: official netmap kernel
Post by: mb on September 16, 2020, 11:20:58 pm
And can I apply this update directly on 20.1.9 or it's better to wait for 20.7.3?

Nope, this is only for 20.7.x releases.
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 16, 2020, 11:21:40 pm
Is it to be expected that now with "20.7.2-netmap" the update functionality proposes to update to "20.7.2"

Yes, that's normal. Because you're not running the standard 20.7.2 kernel.
Title: Re: Call for testing: official netmap kernel
Post by: heresjody on September 16, 2020, 11:22:00 pm
Kernel update worked fine with me.

    OPNsense 20.7.2-amd64
FreeBSD 12.1-RELEASE-p9-HBSD
OpenSSL 1.1.1g 21 Apr 2020

To doublecheck I'm providing valid testing results for you (PPPoE WAN / Suricata)
1. Only select interface: In my case LAN (vtnet) and don't select WAN (vtnet vlan 6)
2. Add public IP to home networks.

Correct?
Title: Re: Call for testing: official netmap kernel
Post by: almodovaris on September 16, 2020, 11:40:47 pm
Nope, it did not change the download speed in APU2. If anything, it got even lower. I use Sensei 1.6 with September definitions.
Title: Re: Call for testing: official netmap kernel
Post by: DanMc85 on September 17, 2020, 12:54:22 am
With the new build, should Interfaces Selection for Protected Interfaces be LAN only or can I add vmx0_VLAN's and ovpns OpenVPN Server interfaces also. Wasn't sure if LAN covered them all or not.
-I do have internal DNS server running on a Domain Controller on the LAN. Not sure if that matters for config.

This is a VMWare ESXi 7 environment if the VMX didn't give it away :)
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 17, 2020, 03:18:01 am
To doublecheck I'm providing valid testing results for you (PPPoE WAN / Suricata)
1. Only select interface: In my case LAN (vtnet) and don't select WAN (vtnet vlan 6)
2. Add public IP to home networks.

Hi @heresjody:

1- Correct. Though we have not touched pppoe+netmap yet. Use Sensei on LAN.
2- Not sure if I understood correctly. Can you elaborate?
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 17, 2020, 03:20:04 am
With the new build, should Interfaces Selection for Protected Interfaces be LAN only or can I add vmx0_VLAN's and ovpns OpenVPN Server interfaces also. Wasn't sure if LAN covered them all or not.
-I do have internal DNS server running on a Domain Controller on the LAN. Not sure if that matters for config.

This is a VMWare ESXi 7 environment if the VMX didn't give it away :)

:)

Yes, do not put Sensei on WAN interface; or any interface that Suricata is also running on.

Otherwise, you can add vmx parent/vlan interfaces. You can also add openvpn interfaces.

If you have an internal DNS, try the new realtime dns mapping feature ;)
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 17, 2020, 03:24:09 am
Nope, it did not change the download speed in APU2. If anything, it got even lower. I use Sensei 1.6 with September definitions.

@almadovaris, APU2 is quite weak. Though, I would not expect BW getting lower. Can you share a few figures?

New netmap kernel + Sensei 1.6 is producing better results for us and for some users we're in touch with. We were able to attain around 1 Gbps  sustained throughput even with old Xeon servers (ubench scores around 180K-220K - all igb drivers).
Title: Re: Call for testing: official netmap kernel
Post by: DanMc85 on September 17, 2020, 03:24:27 am
I only have a free license in a home learning / testing environment but thanks for the info.

Sent from my SM-N986U using Tapatalk

Title: Re: Call for testing: official netmap kernel
Post by: mb on September 17, 2020, 03:25:49 am
I only have a free license in a home learning / testing environment but thanks for the info.

Register for https://sunnyvalley.cloud and you'll find a surprise ;) You can get a trial license for 7 days now - will be publicly announced next week.

Title: Re: Call for testing: official netmap kernel
Post by: DanMc85 on September 17, 2020, 04:37:10 am


I only have a free license in a home learning / testing environment but thanks for the info.

Register for https://sunnyvalley.cloud and you'll find a surprise ;) You can get a trial license for 7 days now - will be publicly announced next week.

Is the trial license the surprise lol

Sent from my SM-N986U using Tapatalk

Title: Re: Call for testing: official netmap kernel
Post by: XeroX on September 17, 2020, 04:03:55 pm
Upgrading on a system with igb drivers gains no improvement or benefit correct?
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 17, 2020, 04:23:33 pm
Upgrading on a system with igb drivers gains no improvement or benefit correct?

Hi @XeroX, with regard to reliability,stock 20.7.2 is already fine. With the beta kernel and 1.6, we've been reported of performance improvements. If you're fine with your performance, you might want to wait for the release.
Title: Re: Call for testing: official netmap kernel
Post by: almodovaris on September 17, 2020, 04:44:30 pm
Nope, it did not change the download speed in APU2. If anything, it got even lower. I use Sensei 1.6 with September definitions.

@almadovaris, APU2 is quite weak. Though, I would not expect BW getting lower. Can you share a few figures?

New netmap kernel + Sensei 1.6 is producing better results for us and for some users we're in touch with. We were able to attain around 1 Gbps  sustained throughput even with old Xeon servers (ubench scores around 180K-220K - all igb drivers).
I am a home user, I won't invest thousand Euros for a router. I have unprotected igb2 (i.e. got it outside of Sensei) and that's where my Usenet download boxes are.

In Opnsense 20.1 I got 180 to 200 Mbps download speed through Sensei.

In Opnsense 20.7 with normal kernel I get 65 to 75 Mbps download speed. With the netmap kernel I get 60 to 65 Mbps.

Of course that's measuring the download speed the same way, with Ookla from my internet provider. If I measure it with Fing for mobile phone I get now (netmap kernel) 120 Mbps to 235 Mbps, depending on the router. Through OpenVPN from a computer behind Sensei the speed is again much higher than without VPN.

I did notice that easpect runs about 92% of one processor core during high speed download.

So, yeah, basically I am content about Sensei since my high speed downloads go outside of it.
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 17, 2020, 05:30:00 pm
Thanks for the further information.

I am a home user, I won't invest thousand Euros for a router. I have unprotected igb2 (i.e. got it outside of Sensei) and that's where my Usenet download boxes are.

Fair enough.

As a side note, for home-use; at a few hundred, it's possible to find some other solutions in the market.
My colleague is able to saturate his 1Gbps AT&T fiber (Sensei 1.6 and 20.7.2-netmap kernel) with a cheap box:
https://www.speedtest.net/result/c/b847d8ab-56a3-42bc-983e-fc6e5d89adfe
Title: Re: Call for testing: official netmap kernel
Post by: almodovaris on September 17, 2020, 05:42:32 pm
Like I said, I got 235 Mbps with Sensei. I suspect that's the maximum my WiFi allows through my mobile phone (I also have other clients for 5 GHz WiFi).

Between 60 Mbps and 235 Mbps there is a huge difference. I don't know how Fing measures speed, but if it would work the same way for Usenet speed, it would be great.

Ok, got it: TLS Usenet connections have a speed of just under 9 MB/s through Sensei. Unencrypted Usenet connections have a speed around 26 MB/s or 27.5 MB/s.

So, unencrypted Usenet reaches about 220 Mbps.
Title: Re: Call for testing: official netmap kernel
Post by: heresjody on September 17, 2020, 10:54:07 pm
To doublecheck I'm providing valid testing results for you (PPPoE WAN / Suricata)
1. Only select interface: In my case LAN (vtnet) and don't select WAN (vtnet vlan 6)
2. Add public IP to home networks.

Hi @heresjody:

1- Correct. Though we have not touched pppoe+netmap yet. Use Sensei on LAN.
2- Not sure if I understood correctly. Can you elaborate?

I’m using Suricata, not Sensei. In the settings of Suricata you can add your public IP to the home network setting. Since PPPoE and netmap hasn’t been touched yet thi setting is irrelevant. For now at least 😬
Title: Re: Call for testing: official netmap kernel
Post by: almodovaris on September 17, 2020, 11:27:21 pm
I have pressed 12 and y without knowing in advance that it will revert my kernel. So, after rebooting APU2, I measured the download speed for unencrypted Usenet through Sensei and it's almost 270 Mbps (instead of 220 Mbps with the netmap kernel).

Found the solution: uncheck "Disable hardware large receive offload" save, and reboot. Then netmap kernel is as fast as stock kernel. I mean for unencrypted Usenet connections I got 275 Mbps out of 300 Mbps nominal Usenet speed. But this does not work for SSL or TLS Usenet connections.
Title: Re: Call for testing: official netmap kernel
Post by: r4nd0m on September 18, 2020, 02:11:10 pm
whats the uname output of the "fixed" kernel?

I did install all available upgrades and ended up with a bootloop so selected previous kernel, disabled sensei 1.6 for the moment at boot while I figure out what I should be seeing ...

this is on ESX 6.7.0 (Build 9484548)
Title: Re: Call for testing: official netmap kernel
Post by: heresjody on September 18, 2020, 02:18:41 pm
FreeBSD 12.1-RELEASE-p9-HBSD
Title: Re: Call for testing: official netmap kernel
Post by: r4nd0m on September 18, 2020, 03:28:42 pm
FreeBSD 12.1-RELEASE-p9-HBSD

so this is the fixed, correct kernel?

FreeBSD host.name.xyz 12.1-RELEASE-p9-HBSD FreeBSD 12.1-RELEASE-p9-HBSD  3b652d8ad0e(master) SMP  amd64
Title: Re: Call for testing: official netmap kernel
Post by: franco on September 18, 2020, 03:31:14 pm
3b652d8ad0e is correct, see https://github.com/opnsense/src/commit/3b652d8ad0e
Title: Re: Call for testing: official netmap kernel
Post by: r4nd0m on September 19, 2020, 05:18:56 pm
ok, re-installed the kernel again, then sensei and its been stable so far - not sure what it was previously
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 19, 2020, 06:05:28 pm
@r4nd0m, thanks for the update.
Title: Re: Call for testing: official netmap kernel
Post by: r4nd0m on September 20, 2020, 05:04:56 pm
@r4nd0m, thanks for the update.

found the issue ... looks like the package does not supersede the original package and will be pulled as an upgrade if upgrades are available leading to the boot loop

I suggest this gets another version number eg 20.7.2.1 so the upgrade wont be suggested back to the non patched kernel

https://i.imgur.com/nk5WZSg.png
Title: Re: Call for testing: official netmap kernel
Post by: nines on September 22, 2020, 03:58:31 pm
Boot loops are fixed for me now however performance went down from 270mbps to 120mbps while using esxi with vmx.

Sensei is disabled, only using ips

Is this known to be an issue?


Sent from iPhone via Tapatalk
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 22, 2020, 04:14:50 pm
Hi @nines, we are able to attain around 1.2-1.3 Gbps between two VMware guests. ubench score around 400K.

What is your HW configuration? What does ubench -cs tell?
Title: Re: Call for testing: official netmap kernel
Post by: nines on September 22, 2020, 04:59:03 pm
Hi @mb thanks for your fast response (as always!)

its an esxi 6.7 running on i5-5300U
https://ark.intel.com/content/www/de/de/ark/products/84862/intel-nuc-board-nuc5i5mybe.html

20.1 ubench
Code: [Select]
./ubench -cs
Unix Benchmark Utility v.0.3
Copyright (C) July, 1999 PhysTech, Inc.
Author: Sergei Viznyuk <sv@phystech.com>
http://www.phystech.com/download/ubench.html
FreeBSD 11.2-RELEASE-p20-HBSD FreeBSD 11.2-RELEASE-p20-HBSD  07ef86ce9ca(stable/20.1) amd64
Ubench Single CPU:   420580 (0.40s)

20.7 ubench
Code: [Select]
./ubench -cs
Unix Benchmark Utility v.0.3
Copyright (C) July, 1999 PhysTech, Inc.
Author: Sergei Viznyuk <sv@phystech.com>
http://www.phystech.com/download/ubench.html
FreeBSD 12.1-RELEASE-p9-HBSD FreeBSD 12.1-RELEASE-p9-HBSD  3b652d8ad0e(master) SMP amd64
Ubench Single CPU:   412479 (0.41s)

any thoughts?
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 22, 2020, 06:30:18 pm
Hi @nines, all welcome.

CPU score looks good. What happens if you just run nmbridge ? (Both Sensei & Suricata off)

Steps (Assuming vmx0 is your interface)

fetch https://updates.sunnyvalley.io/nmbridge/nmbridge
chmod 750 nmbridge
ifconfig vmx0 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
./nmbridge -i netmap:vmx0 -i netmap:vmx0^

Title: Re: Call for testing: official netmap kernel
Post by: nines on September 22, 2020, 07:28:33 pm
what would be the expected output?


Code: [Select]
618.451661 main [247] same interface, endpoint 0 goes to host
618.488299 nm_mmap [990] do not mmap, inherit from parent
618.488349 main [269] ------- zerocopy supported
618.488377 main [276] Wait 4 secs for link to come up...
622.499677 main [280] Ready to go, vmx1 0x1/1 <-> vmx1 0x0/1.

//
If I try to do it between vmx0 and vmx1:
Code: [Select]
./nmbridge -i netmap:vmx0 -i netmap:vmx1 the vm instantly becomes unresponsive and the esx console spams "vmx0 drop mbufs that needs checksum offload"
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 22, 2020, 08:44:14 pm
Hi Nines,

Do not bridge vmx0 and vmx1, to simulate Suricata/Sensei behaviour bridge the ethernet and the hos operating system.

chmod 750 nmbridge
ifconfig vmx0 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
./nmbridge -i netmap:vmx0 -i netmap:vmx0^

Title: Re: Call for testing: official netmap kernel
Post by: nines on September 22, 2020, 08:48:09 pm
Hi mb,

Did that also, it's the first quote in my above post but there is no more output other than this.

// during run of nmbridge no traffic forwarding seems to work, ssh is still working though.
if I control+c traffic forwarding continues to work:

Code: [Select]
^C864.502603 main [331] poll timeout [0] ev 1 0 rx 0@42 tx 443, [1] ev 1 0 rx 0@69 tx 511

Sent from iPhone via Tapatalk
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 22, 2020, 09:01:04 pm
Did that also, it's the first quote in my above post but there is no more output other than this.

// during run of nmbridge no traffic forwarding seems to work, ssh is still working though.

If you're running nmbridge on vmx1, than you need to disable all offloadings on this interface also:

ifconfig vmx1 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
./nmbridge -i netmap:vmx1 -i netmap:vmx1^


Then, run another round of speed test, while nmbridge is running. It is ok that nmbridge won't print much output.
Title: Re: Call for testing: official netmap kernel
Post by: nines on September 22, 2020, 09:09:49 pm
thanks for your patience.
its much better but not as good as with 20.1 - now its fluctuating between 130 and 275mbps while 20.1 always peaked at 275 (max isp speed)
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 22, 2020, 10:34:14 pm
Hi @nines, no worries at all. My pleasure.

If you can see 275 Mbps now; but not always; I'm inclined to think fluctuations might be related to virtual environment. I'd suggest testing while trying to reduce any side effects imposed by other hosts and/or other services on the firewall.

Reason is, in our test environment, our tests show 1.2 Gbps - 1.3 Gbps between two guests. There's some fluctuation, but it's more like 10% or so.
Title: Re: Call for testing: official netmap kernel
Post by: nines on September 22, 2020, 10:43:08 pm
ok, I tested again. Simple http download with disabled suricata/sensei is fast at ~270mbit. If ips is off speeds are max, as soon as its enabled suricata is saturating one core with 100% and speed is capped at 120mbit.

So the reson is definetely something related to ips ...
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 22, 2020, 10:46:28 pm
Got it, thanks for the update.

Title: Re: Call for testing: official netmap kernel
Post by: nines on September 22, 2020, 10:51:22 pm
So I should better go back to 20.1 as you can't reproduce that behavior on your site right?
ruleset is the same so that shouldn't be an issue


Sent from iPhone via Tapatalk
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 23, 2020, 01:02:22 am
Hi @Nines, correct. Netmap-wise, you look good.

WRT IPS, I'd suspect from the configuration & rulesets if you were able to attain higher speeds on 20.1.
Title: Re: Call for testing: official netmap kernel
Post by: nines on September 23, 2020, 11:44:53 am
did some more tests.
its slow if ips is enabled on "wan"
its fast if ips enabled on "lan"
How can that be? What am I missing here? Shouldnt the rules be processed in either scenario? Where does the difference in performance come from?

just for reference:

 
WRT IPS, I'd suspect from the configuration & rulesets if you were able to attain higher speeds on 20.1.

dont know how because the configuration and rulesets did NOT change.
Title: Re: Call for testing: official netmap kernel
Post by: scream on September 23, 2020, 06:07:58 pm
Installed on test vm.
Performance with vmx interfaces are still bad so I have to revert to 20.1.

Title: Re: Call for testing: official netmap kernel
Post by: mb on September 23, 2020, 07:25:50 pm
did some more tests.
its slow if ips is enabled on "wan"
its fast if ips enabled on "lan"
How can that be? What am I missing here? Shouldnt the rules be processed in either scenario? Where does the difference in performance come from?

Hi @Nines, nmbridge test shows the results with regard to the netmap & kernel alone. If this provides the expected result, this means you're good with regard to the kernel & netmap.

For IPS, unfortunately I'm not so much deep into the details. How about discussing it in the IPS forum?
Title: Re: Call for testing: official netmap kernel
Post by: nines on September 23, 2020, 07:59:52 pm
Hi mb, as I wrote in #42 ithe results with nm bridge were better but there was still a huge difference compared to 20.1 WITH enabled ips.
In my oppionion the current kernel is definetely not performing as good as with 20.1 and as such the performance with enables ips is even worse.

Yes, ubench result is good, but overall performance is far away from 20.1. I know you and your team have put a huge amount of work into this but I think it's better to point out potential issues then having to deal with the complaints later on.

To be clear: you can definetely assume that the results mentioned here are not sourced from external factors like esx load, webserver load or something else. I can easily check this by just restoring both vmware snapshots back and forth - it MUST have something to do with 20.7 and/or the kernel. The results are strictly "following the problem atic version" and never appear with 20.1 (as said, performance with 20.1 and enabled ips is always a lot better then 20.7 with disabled ips, rest of the config is exactly the same)

Thanks again for your support and work.


Sent from iPhone via Tapatalk
Title: Re: Call for testing: official netmap kernel
Post by: heresjody on September 24, 2020, 07:15:26 pm
Still going strong with the test kernel. On both my em and my vtnet interface servers no crashes since I installed in 7 days ago. 

Just checking: Can I safely apply the netmap test kernel to 20.7.3?
Title: Re: Call for testing: official netmap kernel
Post by: andreaslink on September 24, 2020, 08:08:17 pm
Kind of a same question, I just updated to v20.7.3 and came from the 20.7.2-netmap kernel. Now my wireguard side2side tunnel does not connect anymore.

Is there something like opnsense-update -kr 20.7.3-netmap so I can proceed testing the new kernel?
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 24, 2020, 08:14:24 pm
Hi @heresjody, @andreas,  it's best to wait for @franco to announce 20.7.3-nemap kernel. It's most probably not there yet.
Title: Re: Call for testing: official netmap kernel
Post by: andreaslink on September 24, 2020, 08:18:54 pm
Upsi, then I was too ambitious....

root@OPNsense:~ # opnsense-update -kr 20.7.3-netmap
Fetching kernel-20.7.3-netmap-amd64.txz: ......... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-20.7.3-netmap-amd64.txz... done
Please reboot.
Title: Re: Call for testing: official netmap kernel
Post by: heresjody on September 24, 2020, 08:20:53 pm
Upsi, then I was too ambitious....

root@OPNsense:~ # opnsense-update -kr 20.7.3-netmap
Fetching kernel-20.7.3-netmap-amd64.txz: ......... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-20.7.3-netmap-amd64.txz... done
Please reboot.

😂😂 But does it work? 🤔
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 24, 2020, 08:26:03 pm
I guess @andreas will make the 20.7.3-netmap announcement soon ;)
Title: Re: Call for testing: official netmap kernel
Post by: andreaslink on September 24, 2020, 08:47:28 pm
Hehe, sorry, that was not my intention, I thought, if no package is available yet, it would not find anyhing and therefor not proceed.
That's why I surprisped myself slightly :P. So as said, I guess too ambigious, it rebooted, but kernel does not look much different (it's from 21.09.).
Title: Re: Call for testing: official netmap kernel
Post by: franco on September 24, 2020, 09:19:56 pm
We put 20.7.3-netmap there for users to find to provide all the updates of 20.7.3 with all the netmap things from master branch plus Realtek driver update. No need to wait for approval.  ;)


Cheers,
Franco
Title: Re: Call for testing: official netmap kernel
Post by: andreaslink on September 24, 2020, 09:52:08 pm
Cool, thanks @Franco, so if I understood you correctly, then I'm back on the netmap testing track with my update now? So the update worked as initially assumed by me?
Title: Re: Call for testing: official netmap kernel
Post by: franco on September 24, 2020, 11:34:02 pm
Yep. Last test kernel, I promise.


Cheers,
Franco
Title: Re: Call for testing: official netmap kernel
Post by: AceRickslick on September 25, 2020, 07:26:19 pm
OPNsense 20.7.3-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1g 21 Apr 2020
AMD Athlon 3000G with Radeon Vega Graphics (4 cores)
8GB RAM and Intel E1G42ET Dual Ethernet
Virgin Gig1 Broadband

When downloading large files via an SSL encrypted connection with Sensei 1.6 in standard Routed Mode and the latest 20.7.3 netmap kernel, my download speed maxes out at about 450mbps, but running Speedtest I get about 930mbps. If I change Sensei to Routed Mode with generic netmap driver, SSL downloads peak at 580mbps. If I switch mode to Passive, then SSL downloads reach max speed about 980mbps.

Doing the same SSL download with 20.1 and Sensei in standard Routed Mode I will max out my broadband speed.

Hoping this info can help improve the netmap kernel.

Thanks for the great work being done!
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 25, 2020, 07:42:15 pm
Hi AceRickslick,

Thanks for the detailed information. What does ubench tell about your cpu score:

Code: [Select]
ubench -cs
Which ethernet adapter are you using for Sensei-protected interfaces and for the WAN interface?

Can you get 930 Mbps with Sensei on during speedtest test?
Title: Re: Call for testing: official netmap kernel
Post by: AceRickslick on September 25, 2020, 09:56:21 pm
Hi mb,

ubench score is 574396.

For Sensei-protected it is igb0 and igb1 is used for WAN. It's an Intel 82576 Chipset ethernet adapter.

Yes, I can get 930 Mbps with Sensei turned on during a speedtest test. Just can't hit those speeds during SSL downloads where before on 20.1 it worked perfectly.
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 25, 2020, 10:22:13 pm
Thanks, that's promising.

Can I ask for one more test: what happens if you put Sensei on bypass mode? This will tell if this is related to Sensei or netmap.
Title: Re: Call for testing: official netmap kernel
Post by: AceRickslick on September 25, 2020, 11:34:47 pm
When Sensei is in Bypass Mode (Passive Mode), I get full speed on SSL downloads and Speedtest test, about 980 Mbps.
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 25, 2020, 11:40:28 pm
Be aware, Passive and Bypass mode are different. Passive mode do not use netmap at all. You can enter Bypass mode via Sensei -> Status -> Enter Bypass Mode

Is it Bypass mode?
Title: Re: Call for testing: official netmap kernel
Post by: AceRickslick on September 26, 2020, 09:18:08 am
My bad it was in Passive Mode, I have retested in Bypass Mode and I get full Gigabit speeds on Speedtest and SSL downloads, about 980 Mbps.

Also just to rule out busy network times/time of day, when I ran the above test, I also tested Sensei in the default Routed Mode, Speedtest hits about 980 Mbps but on the exact same SSL file download as above the speed never gets above 480 Mbps.
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 26, 2020, 08:18:37 pm
Hi @Ace,

No worries at all, and thanks for the update. Now it's clear. It looks like we need an optimization on SSL-based classifier.

I'll update the thread once we have some results.
Title: Re: Call for testing: official netmap kernel
Post by: the-mk on September 26, 2020, 09:02:29 pm
question: when I try the test kernel with that command - "opnsense-update -kr 20.7.3-netmap" and reboot afterwards, how do I identify with "uname -a" if the "right" kernel (the netmap-test-kernel) is used?

currently I get the following:
FreeBSD OPNsenseTEST.test 12.1-RELEASE-p10-HBSD-FreeBSD 12.1-RELEASE-p10-HBSD #0 517e44a00df(stable/20.7)-dirty: Mon Sep 21 16:21:17 CEST 2020  root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64
Title: Re: Call for testing: official netmap kernel
Post by: xpendable on September 27, 2020, 03:08:23 am
I just upgraded to the latest 20.7.3 release from 20.1.9 now that this should be the last netmap test kernel.

I disabled Suricata and Sensei first and then proceeded with the upgrade. The upgrade had no issues and the 20.7.3-Netmap kernel installed successfully as well. I then re-enabled Suricata and Sensei, however after everything was back up and running I noticed that my download bandwidth has been reduced considerably as others have mentioned in previous posts as well. Below are my results including general configuration details.

Host: ESXi 6.7 Update 3 (Build 16713306)
OPNsense VM: 4 vCPUs and 8GB RAM
Physcial CPU: Intel Core i7-4790S
Interfaces: vmx0 (LAN - Sensei), vmx1 (WAN - Suricata), no VLANs
Max Download Speed: 750mbps

OPNsense 20.7.3 with 20.7.3-Netmap Kernel:
Suricata Disabled and Sensei PE Stopped (Native) - 720mbps
Suricata Disabled and Sensei PE Bypassed (Native) - 600mbps
Suricata Disabled and Sensei PE Enabled (Native) - 320mbps
Suricata Enabled (Hyperscan/High Profile) and Sensei PE Stopped (Native) - 440mbps
Suricata Enabled (Hyperscan/High Profile) and Sensei PE Bypassed (Native) - 600mbps
Suricata Enabled (Hyperscan/High Profile) and Sensei PE Enabled (Native) - 420mbps

Odd that with Suricata Disabled the throughput goes down, and with Suricata Enabled with the Sensei packet engine bypassed the throughput actually goes up when compared to the stopped state.

OPNsense 20.1.9:
Suricata Enabled and Sensei Enabled - 720mbps

Speedtest was run 3 times in each configuration using the same speedtest server.
I also tried Suricata with the Medium/High & Custom Profile's with no effect on throughput, so I left it on High.

I tested the "generic/emulated" netmap driver through the Sensei config, and the throughput on that was atrocious by dropping the download speed to 15mbps. This is the only setting that also affected my upload throughput.

With Sensei 1.6 my Wireguard interface does not show up as an available interface to add as a protected interface, this should now be possible with Sensei 1.6 correct?

I noticed that once Suricata starts it uses 1 CPU core @ 100% for about 5mins (After Boot as well), during this time you can not run a speedtest as it affects throughput by about 100-150mbps in my setup. Not sure this is relevant, but figured I would mention it as this is new from my OPNsense 20.1.9 install.
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 27, 2020, 04:07:52 pm
question: when I try the test kernel with that command - "opnsense-update -kr 20.7.3-netmap" and reboot afterwards, how do I identify with "uname -a" if the "right" kernel (the netmap-test-kernel) is used?

Hi @the-mk, mine shows the following

# opnsense-update -kr 20.7.3-netmap
Your system is up to date.
# uname -a
FreeBSD fw.local 12.1-RELEASE-p10-HBSD FreeBSD 12.1-RELEASE-p10-HBSD #1  ebb8c1489c7(master)-dirty: Mon Sep 21 13:50:27 CEST 2020     root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP  amd64
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 27, 2020, 04:09:28 pm
Hi @xpendable, thank you for the detailed test results.

Can you reach out to us via "Report Bug" menu from the Sensei UI? We'd like to have a closer look at your configuration & system.

PS: Make sure you check all options and share relavent information.
Title: Re: Call for testing: official netmap kernel
Post by: athurdent on September 28, 2020, 09:01:19 am
Using Netmap, e.g. by turning on Sensei on LAN or Suricata in IPS mode on WAN, the traffic graph on the dashboard stops working for those interfaces, and Zabbix agent is unable to gather interface statistics, too.
Title: Re: Call for testing: official netmap kernel
Post by: sy on September 28, 2020, 06:21:02 pm
Hi @athurdent,
Traffic Graph problem existed with the early 20.7 release but it isn't anymore with 20.7.2 and 3.What is your OPNsense version?
Title: Re: Call for testing: official netmap kernel
Post by: athurdent on September 28, 2020, 06:42:48 pm
Hi @athurdent,
Traffic Graph problem existed with the early 20.7 release but it isn't anymore with 20.7.2 and 3.What is your OPNsense version?
Odd, I‘m on OPNsense 20.7.3-amd64 with this experimental kernel, double checked. Happens with igb and vtnet here.
Cannot attach my screenshot from the iPad as it is too big it seems, so you have to take my word for it. 😅
Title: Re: Call for testing: official netmap kernel
Post by: gauthig on September 29, 2020, 02:31:57 am
I confirm that I do not get a kernel panic when using vmxnet3 in ESXI and IDS or Sensei and the 20.7.3-netmap kernel.   

But that's all I can confirm is no kernel panic, it is still very slow.  My simple but very trusted/reliable test is to use iperf3 on both the opnsense vm and another local vm on in the same ESXI host.

iperf3 -c 192.168.1.1 -R (using -R to simulate download speed to machine)

With IDS off
[ ID] Interval           Transfer     Bitrate                    Retr
[  5]   0.00-10.01  sec  2.27 GBytes  1.95 Gbits/sec    0

With IDS on
[ ID] Interval           Transfer     Bitrate                    Retr
[  5]   0.00-10.00  sec  1.06 GBytes   915 Mbits/sec  2375


Look at the number of retransmissions.  It's not bound by memory or CPU.  CPU never goes over 20% and memory has many GB's left. 
E3-1285V6 - 4 CPU's allocated and 12GB ram.


Title: Re: Call for testing: official netmap kernel
Post by: mb on September 29, 2020, 02:35:52 am
Hi @gauthgig, thanks for the figures. Couple of questions:

1. By "slow", do you mean 915Mbps or do you have lower speeds than this?
2. How was the situation with OPNsense 20.1.x?
3. How does Sensei bypass mode behave?


Title: Re: Call for testing: official netmap kernel
Post by: gauthig on September 29, 2020, 03:07:42 am
By slow I mean a drop from 1.95Gbs to 0.915Gbs, 50% reduction.
In 20.1.x I was seeing about 1.7Gbs, so much less drop when netmap enabled.

I only showed the Suricata on LAN.  I'll re-run with sensei normal and bypass more and send the results.   By the way, my ELK stack is on another ESXI with a 10Gbs link, so the ELK CPU/Memory load will not impact opnsense/sensei.
Title: Re: Call for testing: official netmap kernel
Post by: mb on September 29, 2020, 03:09:33 am
Thanks, looking forward to Sensei results.

Does anyone know if OPNsense 20.7.3 and 20.1.9 have different Suricata releases, or is it the same major release?
Title: Re: Call for testing: official netmap kernel
Post by: nines on September 29, 2020, 06:19:22 am
@mb as far as I know it's suricata 4x vs suricata 5.x


Sent from iPhone via Tapatalk
Title: Re: Call for testing: official netmap kernel
Post by: J. Lambrecht on September 29, 2020, 03:13:28 pm
By slow I mean a drop from 1.95Gbs to 0.915Gbs, 50% reduction.
In 20.1.x I was seeing about 1.7Gbs, so much less drop when netmap enabled.

I only showed the Suricata on LAN.  I'll re-run with sensei normal and bypass more and send the results.   By the way, my ELK stack is on another ESXI with a 10Gbs link, so the ELK CPU/Memory load will not impact opnsense/sensei.

this is normal for an IDS, it inspects every packet, if you enable all rules this is even optimistic. disabling some rules may show a noticeable performance increase with IDS enabled
Title: Re: Call for testing: official netmap kernel
Post by: gauthig on September 29, 2020, 04:04:49 pm
Good news - I got the 20.7.3-netmap working great with both Suricata (WAN) and Sensei (LANs) working great.

I posted some stats yesterday that showed a 50% drop which never happened in 20.1.x (normally about a 3-5% drop on my hardware).  Here are the new results:

Internal Testing  - controlled no other network noise
iperf3 -c 192.168.1.1 -R (using -R to simulate download speed to machine)

With Sensei off
[ ID] Interval                Transfer         Bitrate            Retr
[  5]   0.00-10.37  sec  4.04 GBytes  3.34 Gbits/sec   0

With Sensei on
[ ID] Interval                Transfer          Bitrate         Retr
[  5]   0.00-10.37  sec  3.92 GBytes  3.14 Gbits/sec  0

Going in or out from the LAN to the internet (Passing both Suricata and Sensei Netmap interfaces)
Speedtest
Both IDS off
Download: 903.26 Mbit/s   Upload: 956.85 Mbit/s
Both IDS on
Download: 891.26 Mbit/s   Upload: 910.85 Mbit/s

What Changed - from yesterday - ESXI, recently I did a upgrade from ESXI6.7 to 7.0.  I found an article about memory delays which a patch is coming for.  In the meantime it was recommended to RESERVE ALL Memory for latency sensitive VMs.   I did that and now netmap is working great.  Looking forward for the test kernel to be rolled into the news production update along with the ESXI patch.   I was even able to drastically reduce the resources available to opnsense, here is what the final test numbers above were on:
E3-1285V6 - 2 cpus, 6G RAM
Title: Re: Call for testing: official netmap kernel
Post by: xpendable on September 30, 2020, 01:57:11 am
Hi @gauthig,

I tried the reserve all memory option for my OPNsense VM in ESXi 6.7, unfortunately it made no difference for my setup. Currently I can not upgrade to ESXi 7 due to hardware compatibility issues. However my OPNsense upgrade was in place and was the only thing that changed in my environment. So I'm guessing that the performance issues are with OPNsense 20.7 and the new netmap kernel.
Title: Re: Call for testing: official netmap kernel
Post by: keanu on October 01, 2020, 05:13:56 pm
Just wanted to add in my thanks, working on my end and has fixed my flakey gig connection ranging from 400-600Mbps to sit solid at a gig!
Title: Re: Call for testing: official netmap kernel
Post by: xpendable on October 06, 2020, 08:41:12 pm
I was able to improve my download bandwidth by increasing the network transmit and receive descriptors, I'm now getting around 600mbps. So not as high when I was on OPNsense 20.1.9, but I'm fairly satisfied with my download speed now. Hopefully there are further improvements in the future that will increase performance even more.

Just as a fyi... I did try a number of VMware ESXi settings such as memory reservation, CPU affinity, CPU reservation and setting the latency of the VM in VM options/advanced to High with no improvements.
Title: Re: Call for testing: official netmap kernel
Post by: xpendable on October 08, 2020, 10:15:50 pm
With the Sensei 1.6.1 update I am now getting around 700mbps due to the SSL/TLS performance improvements. During the test it actually reaches speeds into the 750-780mbps but can not sustain it and drops to around 700mbps. So great work in improving the throughput in Sensei 1.6.1.

I noticed that this increase was only noticed in SSL/TLS connections (HTTPS) as that is what they were meant to address. Can these improvements also be made for unencrypted connections? On an HTTP only speed test the results stay around 530mbps for me.
Title: Re: Call for testing: official netmap kernel
Post by: almodovaris on October 09, 2020, 10:42:20 pm
Yup, agree, Ookla from my ISP now gives 138 Mbps instead of 60 to 75 Mbps. Ookla measures by SSL connections.

Also, there is no longer a large speed difference between unencrypted and encrypted Usenet downloads.
Title: Re: Call for testing: official netmap kernel
Post by: mb on October 09, 2020, 11:53:33 pm
@gauthig, @keanu @ @almodovaris, @xpendable, Thanks for the update and insights.

Glad to hear that TLS/SSL speeds are up. We're also scrutinizing the HTTP processor to see if we can save more cpu cycles.
Title: Re: Call for testing: official netmap kernel
Post by: FullyBorked on October 21, 2020, 12:56:05 am
Using Netmap, e.g. by turning on Sensei on LAN or Suricata in IPS mode on WAN, the traffic graph on the dashboard stops working for those interfaces, and Zabbix agent is unable to gather interface statistics, too.

Curious if there has been any progress in resolving this?  Feels like it's been broken a long time. 
Title: Re: Call for testing: official netmap kernel
Post by: mb on October 21, 2020, 01:06:01 am
Hi @FullyBorked,

It's on our agenda now. It seems to be related to the ordering of bpf(4) processing in iflib(4). One of the first issues we'll have a look following 20.7.4 release.
Title: Re: Call for testing: official netmap kernel
Post by: FullyBorked on October 21, 2020, 01:07:18 am
Hi @FullyBorked,

It's on our agenda now. It seems to be related to the ordering of bpf(4) processing in iflib(4). One of the first issues we'll have a look following 20.7.4 release.

Awesome, thanks for the update and the hard work.   8)
Title: Re: Call for testing: official netmap kernel
Post by: Rickytr on October 26, 2020, 03:13:38 pm
I updated to 20.7.4, which includes the modified netmap, but I still have the bandwidth problem. If I enable Sensei my connection drops from 850Mb to about 420Mb. I'm using em driver on Vmware 6.7.
Title: Re: Call for testing: official netmap kernel
Post by: mb on October 26, 2020, 11:24:56 pm
Hi @RickyTR, can you share the hardware details of the hypervisor? Did you have a chance to try with vmx driver?
Title: Re: Call for testing: official netmap kernel
Post by: Rickytr on October 27, 2020, 01:18:28 pm
Hi @mb. The hardware is 2 CPUs x Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz with 8GB RAM. I don't think the problem may be related to hw performance because with Sensei active the Cpu is around 5% and never goes over 60%. I tried with vmx with the same result.
Title: Re: Call for testing: official netmap kernel
Post by: mb on October 27, 2020, 07:10:56 pm
Hi RickyTR,

Make sure you track cpu usage via "top -SP" and see if any CPU core is fully utilized.

And one more question: are you testing the speed to the internet or between two local LANs?

Title: Re: Call for testing: official netmap kernel
Post by: Rickytr on October 27, 2020, 07:33:23 pm
Hi mb,

top -SP is not showing anything strange, no single CPU is at 100%. I tried testing the speed to the internet.
Title: Re: Call for testing: official netmap kernel
Post by: mb on October 28, 2020, 12:15:41 am
Hi @RickyTR, thanks for more information. Your CPU looks decent. You should be able to saturate a 1 Gig WAN connection.

With a Dual core Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz / OPNsense 20.7.4; I am able to do 930-940Mbps (both Sensei/Suricata on):

https://www.speedtest.net/result/10321829371

Virtualization comes into the scene as a new variable which might change things. Any chances that you can reach out support? Let us run a few tests on your system.
Title: Re: Call for testing: official netmap kernel
Post by: xpendable on October 28, 2020, 05:35:13 pm
Hi @Rickytr,

I had similar problems in ESXi with the vmx adapters. I found that increasing the interfaces transmit and receive descriptors provided a big improvement. You could also try increasing the amount of transmit and receive queues as well.

This option (hw.pci.honor_msi_blacklist="0") is required in order to enable the number of transmit and receive queues to be overridden. Also the number of queues need to be less then or equal to the number of CPU cores assigned to your VM.

These values override the transmit (ntxqs) and receive (nrxqs) queues:
dev.vmx.0.iflib.override_ntxqs="4"
dev.vmx.0.iflib.override_nrxqs="4"

These values override the transmit (ntxds) and receive (nrxds) descriptors:
dev.vmx.0.iflib.override_ntxds="0,2048"
dev.vmx.0.iflib.override_nrxds="0,1024,0"

Since I have 2 vmx interfaces I specify which interface these overrides are applied to:
1st vmx interface (dev.vmx.0) - LAN
2nd vmx interface (dev.vmx.1) - WAN

Create /boot/loader.conf.local if not already created, and add the following and change the ntxqs/nrxqs queue values to match the number of your CPU cores. Also you can adjust the ntxds/nrxds values as needed, but keep in mind that the vmx interface has a maximum value of 4096 for the transmit descriptors and a maximum value of 2048 for the receive descriptors.
Code: [Select]
# VMware tunables for vmx interfaces
hw.pci.honor_msi_blacklist="0"
dev.vmx.0.iflib.override_ntxqs="4"
dev.vmx.0.iflib.override_nrxqs="4"
dev.vmx.0.iflib.override_ntxds="0,2048"
dev.vmx.0.iflib.override_nrxds="0,1024,0"
dev.vmx.1.iflib.override_ntxqs="4"
dev.vmx.1.iflib.override_nrxqs="4"
dev.vmx.1.iflib.override_ntxds="0,2048"
dev.vmx.1.iflib.override_nrxds="0,1024,0"

Remember to take a snapshot of your VM before making any changes, so you have an easy way to recover just in case.

There is a good explanation of these tunables here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237166

The tunables provided in the link below have no affect, however the information on the queues and descriptors are still valid:
https://www.freebsd.org/cgi/man.cgi?query=vmx&sektion=4
Title: Re: Call for testing: official netmap kernel
Post by: GreenMatter on October 29, 2020, 09:44:08 am
I use ESXi 7.0.1 (VMs upgraded to same version) and recently have upgraded from 20.1.9 to 20.7.4. Esxi runs on Supermicro board with Intel NICs (and vmxnet3 in VMs). Sensei is active on vlans parent interface, no Suricata running.
Inside my LAN I'm able to saturate my 1 Gb network:
Code: [Select]
Connecting to host 172.16.1.1, port 59242
[  5] local 172.17.0.2 port 36110 connected to 172.16.1.1 port 59242
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   111 MBytes   932 Mbits/sec   15    704 KBytes       
[  5]   1.00-2.00   sec   109 MBytes   913 Mbits/sec    0    816 KBytes       
[  5]   2.00-3.00   sec   109 MBytes   912 Mbits/sec    0    912 KBytes       
[  5]   3.00-4.00   sec   109 MBytes   912 Mbits/sec    0   1000 KBytes       
[  5]   4.00-5.00   sec   110 MBytes   923 Mbits/sec    0   1.06 MBytes       
[  5]   5.00-6.00   sec   106 MBytes   891 Mbits/sec  235    594 KBytes       
[  5]   6.00-7.00   sec   108 MBytes   902 Mbits/sec   31    554 KBytes       
[  5]   7.00-8.00   sec   109 MBytes   912 Mbits/sec    0    690 KBytes       
[  5]   8.00-9.00   sec   109 MBytes   912 Mbits/sec    0    802 KBytes       
[  5]   9.00-10.00  sec   108 MBytes   902 Mbits/sec    0    899 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.06 GBytes   911 Mbits/sec  281             sender
[  5]   0.00-10.01  sec  1.06 GBytes   908 Mbits/sec                  receiver


iperf Done.
 

And never had a problem to almost saturate my internet connection 300 Mb/s up/down.
But still I use tunable "vmxnet3.netmap_native" as per Re: Call for testing: New netmap enabled kernel (https://forum.opnsense.org/index.php?topic=11477.msg55913#msg55913) Shall I remove it?
Title: Re: Call for testing: official netmap kernel
Post by: nines on October 29, 2020, 10:37:34 am
For me nothing helped but switching to e1000 AND change the ips algorithm to the new keen style variant.

Note: at least in my case, that only seems to work with 20.7.4 and nothing before


Gesendet von iPhone mit Tapatalk
Title: Re: Call for testing: official netmap kernel
Post by: spetrillo on April 10, 2022, 03:53:18 am
Hello all,

I am trying to install Zenarmor onto my firewall. I have several ports(Intel I350 NICs) in a LAG and Zenarmor is saying the driver for the interface(LAG?) is incompatible with Netmap. I am trying to run Zenarmor in passive mode for now. How can I resolve this? I am running 22.1.5 of OPNsense.

Thanks,
Steve