Call for testing: New netmap enabled kernel

[mb]

Dear OPNsense community,

One of the exciting new features, introduced with OPNsense 19.1 release, is the introduction of an alternate test kernel having the latest upstream netmap code.

Netmap is a very important subsystem in the base OS, since it provides the necessary plumbing for the operation of Suricata in IPS mode and also Sensei in particular.

Netmap code in FreeBSD (thus HardenedBSD) was almost 4 years old. It lacked lots of new developments and bug-fixes that have been done in this timeframe.

On the FreeBSD side, we (Sunny Valley Networks) sponsored a development effort to bring the latest upstream netmap code into FreeBSD.

Quite promptly, OPNsense team has now landed all this development to OPNsense 19.1. New functionality can be enabled by switching to the new-netmap-kernel (Instructions below)

As said, new kernel brings lots of bug-fixes and new developments, two of the most notable ones are being:

1 - VirtIO network adapters support:
     You can now run Suricata/Sensei on virtio adapters. Virtio adapters are found mostly on QEMU/KVM based
     Hypervisors like Proxmox, and on Cloud VPS providers.

2 - VLAN child interfaces: You can now run Suricata / Sensei on child vlan interfaces.

There is some more development pending (i.e. native VMware vmxnet support) but as of now and as far as our tests are concerned, we now seem to have a stable netmap implementation.

Bottomline, you should have a more stable Suricata (IPS mode) and Sensei experience after you switch to the new kernel.

Here are the steps for you to run and test the new kernel. Please feel free to share any issues you encountered and we'll do our best to investigate and try to find a solution.

IMPORTANT: Make sure you've completed your upgrade to 19.1. The new kernel is available & compatible with OPNsense 19.1.

To switch to the new-netmap-enabled kernel:

# opnsense-update -bkr 19.1-netmap

After the update & reboot, your 'uname -a' output should be similar: (pay attention to the commit hash and branch, it should be:  c4ec367c3d9(master) )

root@fw:~ # uname -a
FreeBSD fw.local 11.2-RELEASE-p8-HBSD FreeBSD 11.2-RELEASE-p8-HBSD  c4ec367c3d9(master)  amd64


To revert back to the 19.1-default kernel:

# opnsense-update -bkf

Kudos to the OPNsense team for all of their co-operation and help on this.

[lattera]

Once I switch my home network over to a new APU4 next month, I'll give this a shot. Were you looking for a specific time frame in which to conclude the testing?

[mb]

@lattera, thx, that'd be great. As for the deadline,  it'd be cool if we could see the initial status.. maybe in a couple of weeks?

[lattera]

Sounds good. In that case, next chance I get to do network maintenance on the HardenedBSD build infrastructure, I'll test this out.

[jjanzz]

Installed the netmap enabled kernel, seems like it crashes elasticsearch in Sensei constantly. Though, the Sensei service itself is running perfectly. Could it be the case that Sensei is not adjusted yet? Seems I can't activate Sensei on the WAN port - which is a VLAN interface (my provider requires it).

EDIT: rebooted once more (second reboot after kernel installation) and now it seems to work as solid as before.

[mimugmail]

Tested on a VPS driven by OpenStack where it was reproduceable when activating IPS mode all connections were lost with log:

Feb  7 08:57:55 PB-FW1-KARL kernel: 275.156421 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb  7 08:57:55 PB-FW1-KARL kernel: 275.156471 [ 720] netmap_update_config      configuration changed (but fine)
Feb  7 08:57:55 PB-FW1-KARL kernel: 275.161603 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb  7 08:57:55 PB-FW1-KARL kernel: 275.161662 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb  7 08:57:55 PB-FW1-KARL kernel: 275.345680 [1026] netmap_finalize_obj_allocator Unable to create cluster at 100380 for 'netmap_buf' allocator
Feb  7 08:57:56 PB-FW1-KARL kernel: 276.048248 [  79] vtnet_netmap_free_bufs    freed 257 mbufs, 0 netmap bufs on 1 queues
Feb  7 08:57:56 PB-FW1-KARL kernel: 276.238981 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb  7 08:57:56 PB-FW1-KARL kernel: 276.244971 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512


Updated to netmap kernel and now it runs fine, messages above are gone.
Only message I get is when restarting Suricata:

Feb  7 09:19:00 PB-FW1-KARL kernel: 540.160522 [  83] vtnet_free_used           1 sgs dequeued from TX-0 (netmap=0)

But machine stays responsive!
Do you also need tests from machines which were not affected to check general functionality?

[mb]

Installed the netmap enabled kernel, seems like it crashes elasticsearch in Sensei constantly. Though, the Sensei service itself is running perfectly. Could it be the case that Sensei is not adjusted yet? Seems I can't activate Sensei on the WAN port - which is a VLAN interface (my provider requires it).

EDIT: rebooted once more (second reboot after kernel installation) and now it seems to work as solid as before.

Good to hear that @jjanzz, any chances you retained some logs regarding Elastic search issue? Might be a good idea to have a look at them. Normally it shouldn't affect ES.

With regard to Sensei, the only difference is that Sensei will be able to run on VLAN and virtio interfaces.

0.7 intentionally refuses to run on those interfaces, because with old kernel it would just cause traffic flow to cease.

One other note regarding WAN interfaces: Sensei is designed to run on inner-looking interfaces. This is because this way, we can also do a mapping between  userid and local ip. With WAN interfaces we lose this information (because we get packets after they're NAT'd).

Working on Sensei 0.8-beta1, which should arrive soon. This has virtio/VLAN enabled. So that you can fully enjoy the new functionality with the new kernel.

With Suricata, you can just start testing the new kernel on virtio / VLAN interfaces.

To our experience, virtio on QEMU/KVM gives better performance results compared to em.

[mb]

Updated to netmap kernel and now it runs fine, messages above are gone.
Only message I get is when restarting Suricata:

Feb  7 09:19:00 PB-FW1-KARL kernel: 540.160522 [  83] vtnet_free_used           1 sgs dequeued from TX-0 (netmap=0)

But machine stays responsive!
Do you also need tests from machines which were not affected to check general functionality?

@mimugmail, very cool.

That log shouldn't indicate a problem. It's netmap telling that it freed a previously used segment.

Our own tests mostly focused on the new functionality. Actually it would be more helpful if you could also test if any existing functionality got broken.

[Redfish]

Upgraded to the new kernel, rebooted and was met with errors.  Glad to provide any pertinent information.

[mb]

@redfish, thanks. How much memory do you allocate to the VM? Does increasing RAM solve?

[Redfish]

Bare metal with 4GB memory, core i3, and i211 nics.  Apologies for the lack of info in previous post.

[mb]

@redfish, thx for the info.

Do you run Suricata on a single interface or on multiple interfaces? Does the old kernel exhibit the same behavior?

Can you paste the output of this command on both kernels:

# sysctl dev.netmap

[Redfish]

mb, thanks for your inquiries and apologies for the delayed response.  As of now, suricata (set to promiscuous mode) is on a single interface (Lan) which has two vlans.  It will be a day or so before I'll have the chance to get the rest of your requested information.

[mb]

redfish, thx, please take your time. I see there are not many interfaces. Let's have a look at what  dev.netmap sysctl says.

[Redfish]

Current system sysctl dev.netmap output
dev.netmap.ixl_rx_miss_bufs: 0
dev.netmap.ixl_rx_miss: 0
dev.netmap.iflib_rx_miss_bufs: 0
dev.netmap.iflib_rx_miss: 0
dev.netmap.iflib_crcstrip: 1
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size_ 2048
dev.netmap.priv_curr_num:  163840
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 4048
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 200
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 73728
dev.netmap.ring_size: 73728
dev.netmap.priv_if_num: 1
dev.netmap.priv_if_size: 1024
dev.netmap.if_cirr_num: 100
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 1024
dev.netmap.if_size: 1024
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 100000
dev.netmap.admode: 0
dev.netmap.fwd: 0
dev.netmap.flags: 0
dev.netmap.adaptive_io: 0
dev.netmap.txsync_retry: 2
dev.netmap.no_pendintr: 1
dev.netmap.mitigate: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0
dev.netmap.ix_rx_miss_bufs: 0
dev.netmap.ix_rx_miss: 0
devlnetmap.ix_crcstrip: 0