20.7.4 and netmap -- em drivers

[zemsten]

Update was successful but left me with some questions about the netmap features. I was using the 20.7.3-netmap kernel in the last release and it was working excellently. I have a 4-port NIC running the em driver. With the netmap kernel I had suricata running in IPS mode on my WAN and VPN client interfaces, no problems. I also had Sensei running on my parent LAN for a few VLANs in native netmap mode, no problems.

Maybe I'm interpretting your release notes wrong, but it sounded like the netmap kernel was rolled into the default for 20.7.4 and what I'm experiencing leads me to believe it's not. I've got Sensei working okay, although I can't comment on consistency yet. Whenever I enable Suricata though, it starts dropping packets like mad. I looked for a 20.7.4-netmap but don't see it. Am I missing something?

[mb]

Hi zemsten,

You're correct. netmap fixes are now incorporated into the stock kernel and they are shipped with 20.7.4.

Do you see anything weird in suricata logs / var/log/system.log ?

[zemsten]

Excellent, glad we're on the same page about the netmap stuff!

Nothing weird in the suricata logs, which is the first thing I checked. It even starts up with the netmap:em1^ prefix, just like it did on the 20.7.3-netmap kernel. Everything appears totally normal after that. The next log is when I shut it down. I didn't check the system.log, however when I just went to do so, the last entry I have is from Oct 15 which begs another question.... why don't I have current logging happening...? I do ship my logs off to an external syslog server, parsing them with graylog, but I don't see them there either. I'm getting current logs from the components, dhcpd, ntpdate, filterlog mostly. That's what I expect. Hmm. I did go and reset log files via the web GUI and it generated a new one alright and I see the dhcp daemon reload but that's it so far.

[zemsten]

Logging is working fine. Don't know what the hiccup was before. That's fine though. I can successfully run Suricata in IDS mode, so it very much feels like a driver problem to me.

[mb]

Hi @zemsten, I'm not quite sure if I understand completely. Do you still have the problem with Suricata in IPS mode?

[pkirsche]

Hi @all, just found this thread. Unfortunately after upgrading from OPNsense 20.7.3 to OPNsense 20.7.4 I am also seeing massive oacket loss and high latency on my connection.
Before the upgrade I've never had any issues but with this new release, I am receiving just around 300 Mbit of my 1GBit internet line. Also opnsense is showing me a warning on my default gateway (cable box in bridge mode) that I am suffering high latency and packet loss.

Before the upgrade, I always got the full line speed with my box. The setup is still the same, only change was the upgrade to 7.4.
I'm running opnsense on a shuttle DH110 system with an Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz (8 cores).
The network interface is powered by an intel i219LM chipset, named "em0" in opnsense.

I have had IDS/IPS active and running, but also disabled it for testing. without any chances in speed, latency or packetloss.
Is anyone else experiencing issues in this direction or has someone else any idea how to fix this issue?

Help is very very much appriciated!

Thanks a lot.

[zemsten]

Hi @zemsten, I'm not quite sure if I understand completely. Do you still have the problem with Suricata in IPS mode?

Yes, I am still running into the same issue as described in my first post. Right now I just have Suricata in IDS mode on my VPN_WAN and WAN interfaces, simply monitoring traffic rather than intercepting it. As soon as I turn on IPS mode, I have huge packet loss and the VPN gateway goes down all together. I didn't have this problem with the 20.7.3-netmap kernel.

[mb]

Got it. Probably you've already done it, but just to make sure: do you have all offloadings disabled for WAN interface?

[zemsten]

Correct. I disabled all offloading when I first setup IPS with Suricata on 20.7.3-netmap. I just verified that those settings are the same as well.

[mb]

@zemsten, I guess you also had Sensei. If so, can you create a PR through Sensei UI?

Make sure you check all chekboxes - so that we can also have a look at system logs - to see if there's any clue that can lead us to somewhere.

[zemsten]

I have some more testing to do when I get home. I added a bunch of sysctl kernel parameters yesterday just working through optimization for my long fat pipe (4G internet). I want to make sure that I'm causing any more issues before I submit any data.

Looking at the forum today though, it seems there are other issues with IPS floating around, likely unrelated but I'll check into those too.

[zemsten]

Well, I did the testing and nothing seems to have changed unfortunately.

Here's the pertinent excerpt from my loader.conf, for a snapshot of my tuneables.

Code Select Expand


hw.ixl.enable_head_writeback="0"
net.enc.in.ipsec_bpf_mask="2"
net.enc.in.ipsec_filter_mask="2"
net.enc.out.ipsec_bpf_mask="1"
net.enc.out.ipsec_filter_mask="1"
net.inet.icmp.reply_from_interface="1"
net.local.dgram.maxdgram="8192"
vfs.read_max="128"
net.inet.ip.portrange.first="1024"
net.inet.tcp.blackhole="2"
net.inet.udp.blackhole="1"
net.inet.ip.random_id="1"
net.inet.ip.sourceroute="0"
net.inet.ip.accept_sourceroute="0"
net.inet.icmp.log_redirect="0"
net.inet.tcp.drop_synfin="1"
net.inet6.ip6.redirect="1"
net.inet6.ip6.use_tempaddr="0"
net.inet6.ip6.prefer_tempaddr="0"
net.inet.tcp.syncookies="1"
net.inet.tcp.recvspace="65536"
net.inet.tcp.sendspace="65536"
net.inet.tcp.delayed_ack="0"
net.inet.udp.maxdgram="57344"
net.link.bridge.pfil_onlyip="0"
net.link.bridge.pfil_local_phys="0"
net.link.bridge.pfil_member="1"
net.link.bridge.pfil_bridge="0"
net.link.tap.user_open="1"
kern.randompid="347"
net.inet.ip.intr_queue_maxlen="1000"
hw.syscons.kbd_reboot="0"
net.inet.tcp.log_debug="0"
net.inet.icmp.icmplim="0"
net.inet.tcp.tso="0"
net.inet.udp.checksum="1"
kern.ipc.maxsockbuf="4262144"
vm.pmap.pti="0"
hw.ibrs_disable="0"
security.bsd.see_other_gids="0"
security.bsd.see_other_uids="0"
net.inet.ip.redirect="0"
net.inet.icmp.drop_redirect="1"
net.inet.tcp.hostcache.cachelimit="0"
net.inet.tcp.soreceive_stream="1"
net.isr.maxthreads="-1"
net.isr.bindthreads="1"
net.pf.source_nodes_hashsize="1048576"
cc_cubic_load="YES"
net.inet.tcp.cc.algorithm="cubic"
net.link.ifqmaxlen="512"
net.inet.tcp.recvbuf_inc="65536"
net.inet.tcp.recvbuf_max="4194304"
net.inet.tcp.sendbuf_inc="65536"
net.inet.tcp.sendbuf_max="4194304"
net.inet.tcp.mssdflt="1460"
net.inet.tcp.minmss="536"
net.inet.tcp.abc_l_var="44"
net.inet.tcp.initcwnd_segments="44"
net.inet.tcp.rfc6675_pipe="1"
dev.em.0.fc="0"
dev.em.1.fc="0"
dev.em.2.fc="0"
dev.em.3.fc="0"
net.bpf.zerocopy_enable="1"


As I mentioned before, everything was working as expected before, so I'm still not sure what changed....