Is Zenarmor looking into the NETMAP bug?

[DoBoY]

I am one of the ones that even with the new netmap loaded still have random crashes of my vlans when using zenarmor, only way it works is in passive mode. Is sunny valley actively looking into this? is there something we can do to help?

Guess I will need to cancel my service with them for now, no use paying for something I can't use.

I am on latest Opnsense + latest NETMAP Fix, using i226-V intel Nics

[beki]

Hi DoBoY,

Firstly, thanks for your cooperation and all feedback you provided about the netmap case in Forum.

As Franco said in his post (https://forum.opnsense.org/index.php?topic=32114.msg157612#msg157612), igc driver is not compatible with netmap yet.

Did you try to use the emulated netmap mode? Also, please, ensure that vlan hw filtering is disabled.

Zenarmor, FreeBSD, and OPNsense teams have collaborated with Klara Systems to solve the VLAN/LAGG issues related to netmap.
They all have been actively working on it for several weeks and releasing patch kernels after completing their internal tests.

However, driver incompatibility problems, like igc, are not in the scope of this project.
After the current project is accomplished successfully, they are going to start a new project to improve emulated netmap mode.

The main aim of the second project is to solve incompatibility problems for all drivers. It is intended to be able to provide quick support to the new NICs as well.

Lastly, it is needless to say that the success of these projects mostly depends on community support, and feedback from community members, like you, is priceless.

Best

[DoBoY]

Thank you for your reply.

yes I tried emulated, hard crash within a few hours.

Native mode last anywhere from 4-7 days ish before it crashes only my vlans. I can still access my default Vlan1 to reboot firewall or even vpn in for that matter, anythgin on VLAN1 is fine and dandy, in emulated it all went down.

so might be a while to incorporate driver support so for those of us with those NIC's thinks like suricate and zenarmor are not feasible. That's unfortunate as i believe those NIC's are becoming very popular with the home firewall userbase. I specifically went with intel this time because I had to many issue with my realtec NIC's and support in freebsd last time :)

[mb]

Hi @DoBoY,

In our experience, intel is still a good choice in terms of driver compatibility. It's just that igc is fairly new.

The path we'll be following as @beki mentioned, will be a driver-agnostic approach. Roughly almost 95% of the time, the issue is with a new driver introduction (like the case of igc) or a driver update which breaks netmap compatibility.

In that regard, working with OPNsense and Klara teams, we've decided to improve netmap's emulated mode (which works driver agnostic) so that it performs reasonably equally well for all drivers. This work is still under development and hopefully if we can reach our goals, driver compatibility should be a discussion of history.

[BondiBlueBalls]

Hi! Sorry to piggyback on this thread, but have others reported issues with the IXL driver as well? I've experienced the same issue that others have reported while using native netmap on my new server.

If not, add it to the list of incompatible drivers. :) Thanks much, and I'm looking forward to being able to use zenarmor!

[johndchch]

in my experience the best way to deal with freeBSDs intel driver issues ( which are beyond opnsense and zenarmour's control - issue is mainly with Intel themselves ) is simply to virtualise opnsense using esxi and use the vmx drivers - opnsense/zenarmour is solid using vmx ( vmx supports netmap native )

The esxi ix/ixl drivers are super reliable ( as you'd expect ) - i225/i226 is supported by the community driver   

[DoBoY]

in my experience the best way to deal with freeBSDs intel driver issues ( which are beyond opnsense and zenarmour's control - issue is mainly with Intel themselves ) is simply to virtualise opnsense using esxi and use the vmx drivers - opnsense/zenarmour is solid using vmx ( vmx supports netmap native )

The esxi ix/ixl drivers are super reliable ( as you'd expect ) - i225/i226 is supported by the community driver

Yeah maybe so, but I fell like my little FW appliance wold not have enough juice to run esxi and opnsense properly? I have not tested it yet, but its just N5105 system with 16Gb Ram.

[SpinningRust]

Thanks for opening this thread DoBoY and thank you to beki and mb for the honest feedback. I am using the igc(4) driver with the I225-V interfaces on my Protecli box. My problem is well described in Franco's testing thread so no need to describe further here. I have no intention to hijack another thread. :-)

mb/beki: Feel free to PM me if you ever need a tester in the future. I appreciate the approach you will be taking and that it is on the roadmap. Regards.

[SpinningRust]

I was trying out the new netmap2 kernel that franco just released. I did not expect it to resolve zenarmor's issues with netmap for my igc(4) driver, and I confirmed it didn't, but this reminded me of another observation...when I do enable zenarmor in L3 routed mode, my upload bandwidth is significantly reduced. In passive mode I consistently get 15-17Mbps but in L3 routed mode upload is reduced to 2-7Mbps. Download bandwidth remains unaffected at around 940Mbps. It's capped by my 1Gbps switch between the firewall and my computer. Just thought I'd throw that observation out there.

[beki]

Hi SpinningRust,
Could you confirm that this issue occurs in emulated mode?
Best

[ryanhaver]

Native mode last anywhere from 4-7 days ish before it crashes only my vlans. I can still access my default Vlan1 to reboot firewall or even vpn in for that matter, anythgin on VLAN1 is fine and dandy, in emulated it all went down.

I'm in the same boat, but mine crashes within hours. I'm glad I found your thread though because I was going crazy thinking I had done something wrong on my end for my new router deployment.

...when I do enable zenarmor in L3 routed mode, my upload bandwidth is significantly reduced.

I am seeing the same on my end. It doesn't matter if bypass mode is enabled or not. I am forced to disable the packet inspection engine to get my upload speeds back.

Could you confirm that this issue occurs in emulated mode?

I still see the issue in emulated mode.

In our experience, intel is still a good choice in terms of driver compatibility. It's just that igc is fairly new.

Unfortunately, I see issues regardless of the Intel NIC/driver combination that I test with (see below for all the NICs in my system. I will test my Chelsio NICs next and hope that they prove to be more reliable for my setup. To be clear I have not tested with the new netmap2 kernel yet, but I'm willing to give it a shot when I find some time.

root@OPNsense:~ # sysctl -a | grep -E 'dev.(igb|ix|em|ice|cxl).*.%desc:'
dev.ice.1.%desc: Intel(R) Ethernet Connection E823-C for SFP - 1.34.2-k
dev.ice.0.%desc: Intel(R) Ethernet Connection E823-C for SFP - 1.34.2-k
dev.cxl.3.%desc: port 3
dev.cxl.2.%desc: port 2
dev.cxl.1.%desc: port 1
dev.cxl.0.%desc: port 0
dev.igb.3.%desc: Intel(R) I350 (Copper)
dev.igb.2.%desc: Intel(R) I350 (Copper)
dev.igb.1.%desc: Intel(R) I350 (Copper)
dev.igb.0.%desc: Intel(R) I350 (Copper)
dev.ix.1.%desc: Intel(R) X550-T2
dev.ix.0.%desc: Intel(R) X550-T2

[SpinningRust]

Hi SpinningRust,
Could you confirm that this issue occurs in emulated mode?
Best

Yes, the issue definitely does occur in emulated mode. I reported my logs in the other testing thread that franco setup.

[SpinningRust]

However, driver incompatibility problems, like igc, are not in the scope of this project.
After the current project is accomplished successfully, they are going to start a new project to improve emulated netmap mode.

The main aim of the second project is to solve incompatibility problems for all drivers. It is intended to be able to provide quick support to the new NICs as well.

Lastly, it is needless to say that the success of these projects mostly depends on community support, and feedback from community members, like you, is priceless.

Best

I'm not sure exactly when this occurred but sometime after I installed 23.7 or zenarmor 1.14 (started with 23.7.3 as I waited awhile and now on 23.7.4 and updated to 1.14/1.15 zenarmor), my issues vanished with the emulated driver. Everything seems to be working as designed with netmap. Performance is great and policies are working as they should. Traffic is actually being blocked now and netmap isn't crashing. Really excited that this is working as it should. I do not currently have vlan tags, so I can't verify if that is an issue. That was something I eliminated earlier in my troubleshooting process.

Running Protectli VP2420 with Intel I225-V 2.5G NICs and igc driver.