OPNsense Forum

English Forums => 19.1 Legacy Series => Topic started by: mb on February 06, 2019, 12:21:44 am

Title: Call for testing: New netmap enabled kernel
Post by: mb on February 06, 2019, 12:21:44 am
Dear OPNsense community,

One of the exciting new features, introduced with OPNsense 19.1 release, is the introduction of an alternate test kernel having the latest upstream netmap code.

Netmap is a very important subsystem in the base OS, since it provides the necessary plumbing for the operation of Suricata in IPS mode and also Sensei in particular.

Netmap code in FreeBSD (thus HardenedBSD) was almost 4 years old. It lacked lots of new developments and bug-fixes that have been done in this timeframe.

On the FreeBSD side, we (Sunny Valley Networks) sponsored a development effort to bring the latest upstream netmap code into FreeBSD.

Quite promptly, OPNsense team has now landed all this development to OPNsense 19.1. New functionality can be enabled by switching to the new-netmap-kernel (Instructions below)

As said, new kernel brings lots of bug-fixes and new developments, two of the most notable ones are being:

1 - VirtIO network adapters support:
     You can now run Suricata/Sensei on virtio adapters. Virtio adapters are found mostly on QEMU/KVM based
     Hypervisors like Proxmox, and on Cloud VPS providers.

2 - VLAN child interfaces: You can now run Suricata / Sensei on child vlan interfaces.

There is some more development pending (i.e. native VMware vmxnet support) but as of now and as far as our tests are concerned, we now seem to have a stable netmap implementation.

Bottomline, you should have a more stable Suricata (IPS mode) and Sensei experience after you switch to the new kernel.

Here are the steps for you to run and test the new kernel. Please feel free to share any issues you encountered and we'll do our best to investigate and try to find a solution.

IMPORTANT: Make sure you've completed your upgrade to 19.1. The new kernel is available & compatible with OPNsense 19.1.

To switch to the new-netmap-enabled kernel:

# opnsense-update -bkr 19.1-netmap

After the update & reboot, your 'uname -a' output should be similar: (pay attention to the commit hash and branch, it should be:  c4ec367c3d9(master) )

root@fw:~ # uname -a
FreeBSD fw.local 11.2-RELEASE-p8-HBSD FreeBSD 11.2-RELEASE-p8-HBSD  c4ec367c3d9(master)  amd64


To revert back to the 19.1-default kernel:

# opnsense-update -bkf

Kudos to the OPNsense team for all of their co-operation and help on this.
Title: Re: Call for testing: New netmap enabled kernel
Post by: lattera on February 06, 2019, 06:20:52 pm
Once I switch my home network over to a new APU4 next month, I'll give this a shot. Were you looking for a specific time frame in which to conclude the testing?
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 07, 2019, 02:48:02 am
@lattera, thx, that'd be great. As for the deadline,  it'd be cool if we could see the initial status.. maybe in a couple of weeks?
Title: Re: Call for testing: New netmap enabled kernel
Post by: lattera on February 07, 2019, 04:14:03 am
Sounds good. In that case, next chance I get to do network maintenance on the HardenedBSD build infrastructure, I'll test this out.
Title: Re: Call for testing: New netmap enabled kernel
Post by: jjanzz on February 07, 2019, 08:45:32 am
Installed the netmap enabled kernel, seems like it crashes elasticsearch in Sensei constantly. Though, the Sensei service itself is running perfectly. Could it be the case that Sensei is not adjusted yet? Seems I can't activate Sensei on the WAN port - which is a VLAN interface (my provider requires it).

EDIT: rebooted once more (second reboot after kernel installation) and now it seems to work as solid as before.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mimugmail on February 07, 2019, 09:21:19 am
Tested on a VPS driven by OpenStack where it was reproduceable when activating IPS mode all connections were lost with log:

Feb  7 08:57:55 PB-FW1-KARL kernel: 275.156421 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb  7 08:57:55 PB-FW1-KARL kernel: 275.156471 [ 720] netmap_update_config      configuration changed (but fine)
Feb  7 08:57:55 PB-FW1-KARL kernel: 275.161603 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb  7 08:57:55 PB-FW1-KARL kernel: 275.161662 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb  7 08:57:55 PB-FW1-KARL kernel: 275.345680 [1026] netmap_finalize_obj_allocator Unable to create cluster at 100380 for 'netmap_buf' allocator
Feb  7 08:57:56 PB-FW1-KARL kernel: 276.048248 [  79] vtnet_netmap_free_bufs    freed 257 mbufs, 0 netmap bufs on 1 queues
Feb  7 08:57:56 PB-FW1-KARL kernel: 276.238981 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb  7 08:57:56 PB-FW1-KARL kernel: 276.244971 [ 401] vtnet_netmap_config       vtnet config txq=1, txd=512 rxq=1, rxd=512


Updated to netmap kernel and now it runs fine, messages above are gone.
Only message I get is when restarting Suricata:

Feb  7 09:19:00 PB-FW1-KARL kernel: 540.160522 [  83] vtnet_free_used           1 sgs dequeued from TX-0 (netmap=0)

But machine stays responsive!
Do you also need tests from machines which were not affected to check general functionality?
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 07, 2019, 05:36:44 pm
Installed the netmap enabled kernel, seems like it crashes elasticsearch in Sensei constantly. Though, the Sensei service itself is running perfectly. Could it be the case that Sensei is not adjusted yet? Seems I can't activate Sensei on the WAN port - which is a VLAN interface (my provider requires it).

EDIT: rebooted once more (second reboot after kernel installation) and now it seems to work as solid as before.

Good to hear that @jjanzz, any chances you retained some logs regarding Elastic search issue? Might be a good idea to have a look at them. Normally it shouldn't affect ES.

With regard to Sensei, the only difference is that Sensei will be able to run on VLAN and virtio interfaces.

0.7 intentionally refuses to run on those interfaces, because with old kernel it would just cause traffic flow to cease.

One other note regarding WAN interfaces: Sensei is designed to run on inner-looking interfaces. This is because this way, we can also do a mapping between  userid and local ip. With WAN interfaces we lose this information (because we get packets after they're NAT'd).

Working on Sensei 0.8-beta1, which should arrive soon. This has virtio/VLAN enabled. So that you can fully enjoy the new functionality with the new kernel.

With Suricata, you can just start testing the new kernel on virtio / VLAN interfaces.

To our experience, virtio on QEMU/KVM gives better performance results compared to em.

Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 07, 2019, 05:40:06 pm
Updated to netmap kernel and now it runs fine, messages above are gone.
Only message I get is when restarting Suricata:

Feb  7 09:19:00 PB-FW1-KARL kernel: 540.160522 [  83] vtnet_free_used           1 sgs dequeued from TX-0 (netmap=0)

But machine stays responsive!
Do you also need tests from machines which were not affected to check general functionality?

@mimugmail, very cool.

That log shouldn't indicate a problem. It's netmap telling that it freed a previously used segment.

Our own tests mostly focused on the new functionality. Actually it would be more helpful if you could also test if any existing functionality got broken.
Title: Re: Call for testing: New netmap enabled kernel
Post by: Redfish on February 07, 2019, 09:09:53 pm
Upgraded to the new kernel, rebooted and was met with errors.  Glad to provide any pertinent information.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 07, 2019, 10:54:28 pm
@redfish, thanks. How much memory do you allocate to the VM? Does increasing RAM solve?
Title: Re: Call for testing: New netmap enabled kernel
Post by: Redfish on February 07, 2019, 11:26:50 pm
Bare metal with 4GB memory, core i3, and i211 nics.  Apologies for the lack of info in previous post.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 09, 2019, 02:24:49 am
@redfish, thx for the info.

Do you run Suricata on a single interface or on multiple interfaces? Does the old kernel exhibit the same behavior?

Can you paste the output of this command on both kernels:

# sysctl dev.netmap
Title: Re: Call for testing: New netmap enabled kernel
Post by: Redfish on February 09, 2019, 11:20:10 pm
mb, thanks for your inquiries and apologies for the delayed response.  As of now, suricata (set to promiscuous mode) is on a single interface (Lan) which has two vlans.  It will be a day or so before I’ll have the chance to get the rest of your requested information.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 09, 2019, 11:58:51 pm
redfish, thx, please take your time. I see there are not many interfaces. Let's have a look at what  dev.netmap sysctl says.
Title: Re: Call for testing: New netmap enabled kernel
Post by: Redfish on February 10, 2019, 01:18:50 am
Current system sysctl dev.netmap output
dev.netmap.ixl_rx_miss_bufs: 0
dev.netmap.ixl_rx_miss: 0
dev.netmap.iflib_rx_miss_bufs: 0
dev.netmap.iflib_rx_miss: 0
dev.netmap.iflib_crcstrip: 1
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size_ 2048
dev.netmap.priv_curr_num:  163840
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 4048
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 200
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 73728
dev.netmap.ring_size: 73728
dev.netmap.priv_if_num: 1
dev.netmap.priv_if_size: 1024
dev.netmap.if_cirr_num: 100
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 1024
dev.netmap.if_size: 1024
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 100000
dev.netmap.admode: 0
dev.netmap.fwd: 0
dev.netmap.flags: 0
dev.netmap.adaptive_io: 0
dev.netmap.txsync_retry: 2
dev.netmap.no_pendintr: 1
dev.netmap.mitigate: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0
dev.netmap.ix_rx_miss_bufs: 0
dev.netmap.ix_rx_miss: 0
devlnetmap.ix_crcstrip: 0
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on February 11, 2019, 08:58:59 am
Thanks to Murat and all of Sunny Valley for providing this rework and helping with a related Suricata incompatibility!

So far this is exciting *and* promising with good overall stability. :)

Ideally, this would be merged into 19.1.x in March or April, although the core team has not yet decided on a timeline.


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: lattera on February 11, 2019, 07:14:36 pm
Installation of the test world+kernel sets on the APU4c4 securing HardenedBSD's build network and infrastructure succeeded without issue. I've got Suricata running in IPS mode and only inspecting traffic on the LAN interface. Seems to be working fine for me for now. I'll be placing the firewall under load over the next few days and will report back.
Title: Re: Call for testing: New netmap enabled kernel
Post by: lattera on February 12, 2019, 02:52:56 pm
Deployed successfully on another APU4c4 in a lab environment with tunneled IPv6. Working great!
Title: Re: Call for testing: New netmap enabled kernel
Post by: newsense on February 12, 2019, 04:03:11 pm
Does 19.1-netmap kernel revert anything from 19.1.1 and how will it be kept on par with the upcoming 19.1.2+ versions until it is merged into mainline ?

Just trying to assess the security risks.
Title: Re: Call for testing: New netmap enabled kernel
Post by: newsense on February 12, 2019, 04:15:11 pm
Also noticed this swap message after reboot:
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 12, 2019, 07:46:57 pm
@redfish, this looks like something that needs to be investigated on the Suricata side. Will let you know once I have some more time for this.

@newsense, the only difference between 19.1 default kernel and 19.1-netmap kernel should be the netmap subsystem.

Any chances that you boot with the default kernel and try to see if it produces the same message?

@franco, that's great to hear it'll land on the production branch soon.

@lattera, thanks for the tests. Glad to hear that you had no issues.
Title: Re: Call for testing: New netmap enabled kernel
Post by: lattera on February 12, 2019, 07:51:30 pm
One thing I did notice was that suricata seems to go crazy when in IPS mode and configured to monitor gif interfaces. Brings the entire network stack down. I'm gonna guess that netmap doesn't work with tunneling interfaces (at least, not yet?) Thank goodness for the serial console port on the APU devices. :)
Title: Re: Call for testing: New netmap enabled kernel
Post by: Redfish on February 12, 2019, 10:22:37 pm
Thanks mb, here are the outputs for both kernels:
Edit: also need to note that I’ve followed this guide https://forum.opnsense.org/index.php?topic=6590.0 and disabled VLAN_HWTAGGING in order for suricata to function with my vlans.

Stock Kernel

dev.netmap.ixl_rx_miss_bufs: 0
dev.netmap.ixl_rx_miss: 0
dev.netmap.iflib_rx_miss_bufs: 0
dev.netmap.iflib_rx_miss: 0
dev.netmap.iflib_crcstrip: 1
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size: 2048
dev.netmap.buf_curr_num: 163840
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 2048
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 200
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 73728
dev.netmap.ring_size: 73728
dev.netmap.priv_if_num: 1
dev.netmap.priv_if_size: 1024
dev.netmap.if_curr_num: 100
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 1024
dev.netmap.if_size: 1024
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 100000
dev.netmap.admode: 0
dev.netmap.fwd: 0
dev.netmap.flags: 0
dev.netmap.adaptive_io: 0
dev.netmap.txsync_retry: 2
dev.netmap.no_pendintr: 1
dev.netmap.mitigate: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0
dev.netmap.ix_rx_miss_bufs: 0
dev.netmap.ix_rx_miss: 0
dev.netmap.ix_crcstrip: 0

New netmap kernel

dev.netmap.ixl_rx_miss_bufs: 0
dev.netmap.ixl_rx_miss: 0
dev.netmap.iflib_rx_miss_bufs: 0
dev.netmap.iflib_rx_miss: 0
dev.netmap.iflib_crcstrip: 1
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size: 2048
dev.netmap.buf_curr_num: 0
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 0
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 0
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 0
dev.netmap.ring_size: 36864
dev.netmap.priv_if_num: 2
dev.netmap.priv_if_size: 1024
dev.netmap.if_curr_num: 0
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 0
dev.netmap.if_size: 1024
dev.netmap.ptnet_vnet_hdr: 1
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 100000
dev.netmap.generic_hwcsum: 0
dev.netmap.admode: 0
dev.netmap.fwd: 0
dev.netmap.txsync_retry: 2
dev.netmap.mitigate: 1
dev.netmap.no_pendintr: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0
dev.netmap.ix_rx_miss_bufs: 0
dev.netmap.ix_rx_miss: 0
dev.netmap.ix_crcstrip: 0
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 13, 2019, 02:31:55 am
@redfish, thanks. Looks like it's on the Suricata side. I'll have a closer look.
Title: Re: Call for testing: New netmap enabled kernel
Post by: TheGrandWazoo on February 13, 2019, 07:05:47 pm
This is "Excellent" news! ;D. I run a lot of VM with VirtIO on Proxmox VE and have not been able to take advantage of Suricata. Will start converting and testing my OPNSense kernels.

Was very happy with my transition from pfSense to OPNSense, this makes it even better.

Thank you.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 14, 2019, 03:23:42 am
TheGrandWazoo, great that you found this useful :) Please share your experience with Suricata + new kernel.
Title: Re: Call for testing: New netmap enabled kernel
Post by: Antaris on February 14, 2019, 10:37:29 pm
Now when there is 19.1.1 can we use it with netmap enabled kernel and what is the command?

May be "# opnsense-update -bkr 19.1.1-netmap" ?
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 14, 2019, 10:50:46 pm
Antaris, yes, correct.  From the first post in the thread:

To switch to the new-netmap-enabled kernel:

# opnsense-update -bkr 19.1-netmap

After the update & reboot, your 'uname -a' output should be similar: (pay attention to the commit hash and branch, it should be:  c4ec367c3d9(master) )

root@fw:~ # uname -a
FreeBSD fw.local 11.2-RELEASE-p8-HBSD FreeBSD 11.2-RELEASE-p8-HBSD  c4ec367c3d9(master)  amd64


To revert back to the 19.1-default kernel:

# opnsense-update -bkf

Franco plans the merge in the upcoming OPNsense 19.1.x minor releases.
Title: Re: Call for testing: New netmap enabled kernel
Post by: Antaris on February 15, 2019, 05:59:21 am
So is there 19.1.1 netmap enabled or just 19.1 ?
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on February 15, 2019, 03:58:11 pm
There is no base/kernel for 19.1.1 so there is no / won't be a 19.1.1-netmap.

And when 19.1.2 is out with a new base/kernel, then the netmap kernel will be automatically removed as it is a test kernel, but you can lock it from the GUI if you do not want that.


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: donatom3 on February 16, 2019, 07:45:22 pm
So far 3 days on the new netmap kernel and I haven't had an issue. 3 interfaces two of them are vlans on i350s. Before this I was having a lockup that killed connections to the interfaces that sensei was monitoring that required a complete reboot to get connectivity back.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 18, 2019, 06:10:05 pm
donatom3, that's great to hear. Thanks for sharing your results.
Title: Re: Call for testing: New netmap enabled kernel
Post by: oneplane on February 20, 2019, 11:40:02 pm
Any interest in reports on doing this on AWS EC2? I have a few self-bootstrapped OPNSense setups there that work fine. I can just snapshot and try the kernel to see how that turns out.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 20, 2019, 11:59:37 pm
oneplane, definitely. Thanks for raising this.

We want both OPNsense + Suricata and Sensei to be able to work on all cloud providers including AWS. What's your experience up until now?
Title: Re: Call for testing: New netmap enabled kernel
Post by: Redfish on February 21, 2019, 01:03:46 pm
mb, reverted to stock kernel and suricata is working as intended.  Please let me know if there’s anything I can do to help with your investigations.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on February 21, 2019, 06:36:00 pm
@redfish, any chances doing a few more tests for me? This might isolate the situation a bit more. You can send the results through a private message.

First download nmbridge application.

# fetch https://updates.sunnyvalley.io/nmbridge/nmbridge
# chmod +x nmbridge


This is a very minimal application which does bridging. It's based on netmap. I'd like to see if this also produces the same result:

With the new kernel, try to see if this command produces the same message. Replace ix0 to suit your interface naming. Do this also for the other network interfaces installed on the host.

# ./nmbridge -i netmap:ix0 -i netmap:ix0^

When you run the application make sure to see packets are flowing and your network connectivity is intact for that specific interface.



Title: Re: Call for testing: New netmap enabled kernel
Post by: TheGrandWazoo on March 01, 2019, 04:12:29 pm
Good day,

I have had NO issues with Suricata IPS in a HA environment using VIRTIO interfaces. I have been running HAProxy environments, SBC (VoIP) environments (A plugin I am working on) and regular firewall environments. Thank you very much!!!!
With that being said 19.1.2 is now out and it has an "OS" update associated with it. Does this 19.1.2 update include this patch set? If not, is there a new kernel that we need to perform the 'opnsense-update'?

Thanks again,
Kev
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on March 01, 2019, 04:24:05 pm
Hi Kev,

There is no refreshed netmap rework kernel yet. If you want to keep it lock the kernel package from system: firmware: packages tab and perform the update as usual.

Look for the release notes, we'll announce the netmap rework there...


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: TheGrandWazoo on March 04, 2019, 06:40:40 pm
Hi Kev,

There is no refreshed netmap rework kernel yet. If you want to keep it lock the kernel package from system: firmware: packages tab and perform the update as usual.

Look for the release notes, we'll announce the netmap rework there...


Cheers,
Franco

Franco,

Thanks for the info. That is what I thought I should do but never hurts to ask. Everything updated without issues.

Thanks again,

Kev
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on March 17, 2019, 12:18:50 pm
There's a new netmap test kernel out there:

# uname -a
FreeBSD sensey.homeoffice.local 11.2-RELEASE-p9-HBSD FreeBSD 11.2-RELEASE-p9-HBSD  4ea457eb7b8(master)  amd64

To upgrade use:

# opnsense-update -fbkr 19.1.4-netmap
# opnsense-shell reboot

To revert back to the 19.1.4 default kernel:

# opnsense-update -bkf

The old test sets for 19.1-netmap will be removed from the mirrors now. Changes in the new version include:

o netmap: fix crash with monitors and VALE ports
o ixl: remove unnecessary limitations related to netmap
o netmap: small cleanup on em, lem, igb, ixgbe
o netmap: fix knote() argument to match the mutex state
o netmap: improvements to the netmap kloop (CSB mode)
o netmap: add notifications on kloop stop
o netmap: upgrade sync-kloop support
o netmap: refactor logging macros and pipes
o vmx(4): add native netmap support
o netmap: fix lock order reversal related to kqueue usage
o netmap: remove redundant call to nm_set_native_flags()
o all fixes of the 19.1.2 and 19.1.4 operating system updates


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: datiscum on March 18, 2019, 11:34:27 am
After installing the new netmap-kernel, the Intel network card was no longer found. I ran OPNSense under KVM and passed 2 ports of a hardware Intel card into the VM. The third E1000 is a KVM emulated card. With the new kernel only the virtual card is found. Is there a solution for this ?
Standard kernel: pciconf -l
em0@pci0:3:0:0: class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
em1@pci0:4:0:0: class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00

em2@pci0:8:1:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00

netmap kernel:
em0@pci0:8:1:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00

Thank you. I'll take care of it,
  Sasha
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on March 19, 2019, 08:00:37 am
Hi Sasha,

This is very odd as the patches are not supposed to do this.

Is this the case with a normal 19.1.4 as well?


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: datiscum on March 19, 2019, 11:18:03 pm
Hello,

In 19.1.4 with the default kernel everything is OK

Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on March 20, 2019, 10:13:36 pm
Strange, I went through the commits and saw nothing out of the ordinary. Compared to 19.1-netmap kernel was it the same or working there? I know this might sound strange but what happens if you reinstall 19.1.4-netmap?


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: datiscum on March 20, 2019, 11:02:24 pm
I've done that before, but it's always been the same result.

I don't have much experience with FreeBSD, I usually only use Linux and Windows. I would have liked to use the VirtIO network driver under KVM Q35. This works under the standard kernel only with the i440FX chipset from KVM. If anyone has an idea, I would like to try it out.
Title: Re: Call for testing: New netmap enabled kernel
Post by: donatom3 on March 24, 2019, 03:27:24 pm
With the new net map kernel sense pi doesn’t detect any traffic on my interface that isn’t a vlan, but it has no issues with the two interfaces that are vlans. This is in intel i211 nics.

I am also not getting any hits in suricata on that non vlan interface.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on March 24, 2019, 04:42:24 pm
Hi donatom3,

Is the non-vlan interface the trunk (parent) interface for the other vlan child interfaces?
Title: Re: Call for testing: New netmap enabled kernel
Post by: donatom3 on March 26, 2019, 07:57:38 am
MB,

No the vlans were moved to their own interface.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on March 27, 2019, 02:20:55 am
Ok, let's have a look at this together.
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on March 27, 2019, 02:38:23 am
This kernel has also support for native netmap support for vmx(4), VMware VMXNET3 Virtual Interface Controller   device.

https://svnweb.freebsd.org/base?view=revision&revision=344272

Native netmap support should yield better performance compared to the emulated driver.

Much appreciated if someone with an existing VMware deployment could test & provide feedback.

PS: Please note that you'll need to set vmxnet3.netmap_native tunable to 1  (from System: Settings: Tunables) to enable native netmap mode.


Title: Re: Call for testing: New netmap enabled kernel
Post by: Antaris on April 01, 2019, 04:57:13 pm
Do setting vmxnet3.netmap_native require reboot to take effect?
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on April 03, 2019, 10:43:14 am
Hmm, where are we on this now?

We have conflicting reports on this second netmap kernel and the original merge window is approaching now (April/May) so I see potential for delay...


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on April 03, 2019, 06:06:08 pm
Do setting vmxnet3.netmap_native require reboot to take effect?

Hi Antaris,

Yes, you'll need a reboot. Make sure it's in effect. Use this command:

Code: [Select]
sysctl vmxnet3.netmap_native
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on April 03, 2019, 06:07:40 pm
Hmm, where are we on this now?

We have conflicting reports on this second netmap kernel and the original merge window is approaching now (April/May) so I see potential for delay...


Cheers,
Franco

Hi Franco,

Agreed. We need to check a few issues reported if they are to be addressed or not.
Title: Re: Call for testing: New netmap enabled kernel
Post by: Antaris on April 03, 2019, 10:35:41 pm
Do setting vmxnet3.netmap_native require reboot to take effect?

Hi Antaris,

Yes, you'll need a reboot. Make sure it's in effect. Use this command:

Code: [Select]
sysctl vmxnet3.netmap_native

And the result:

sysctl: unknown oid 'vmxnet3.netmap_native'

I added "vmxnet3.netmap_native" in System>>Settings>>Tunables and set it to "1"
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on April 04, 2019, 12:06:57 am
Quote
I added "vmxnet3.netmap_native" in System>>Settings>>Tunables and set it to "1"

Antaris, by checking your configuration, looks like you you're using native netmap for vmx now.

Let me see why tunable does not show up.

Title: Re: Call for testing: New netmap enabled kernel
Post by: lrosenman on April 04, 2019, 02:06:53 pm
I'm getting:
Code: [Select]
root@home-fw:~ # opnsense-update -bkr 19.1-netmap
Fetching base-19.1-netmap-amd64.txz: .. failed, no signature found
root@home-fw:~ # uname -a
FreeBSD home-fw.lerctr.org 11.2-RELEASE-p9-HBSD FreeBSD 11.2-RELEASE-p9-HBSD  f083bc4f8a0(stable/19.1)  amd64
root@home-fw:~ #
[code]

Is this now broken?
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on April 05, 2019, 12:01:35 am
Hi Irosenman,

You should now use:

Code: [Select]
opnsense-update -fbkr 19.1.4-netmap
See: https://forum.opnsense.org/index.php?topic=11477.msg55261#msg55261
Title: Re: Call for testing: New netmap enabled kernel
Post by: lrosenman on April 05, 2019, 03:16:21 am
should this help on em(4) devices with normal speed test type stuff? 

I'm on a protectli (https://www.protectli.com) FW1
on ATT Fiber (using https://github.com/aus/pfatt bypass for the RG).

My speeds are ~600meg/sec, but with the Ubiquiti USG I was getting 900+.

Ideas welcome...

I didn't make any changes to the config other than adding the new netmap kernel.

Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on April 05, 2019, 10:44:00 am
I don't expect radical changes on em drivers. There's one commit we don't have yet which would suggest a considerable speedup in the IPS department, however:

https://svnweb.freebsd.org/base?view=revision&revision=345269

What remains to be seen is if this requires changes to Suricata to make use of the speedup potential.



Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: lrosenman on April 05, 2019, 12:33:19 pm
Should I take my performance issue to another topic?
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on April 05, 2019, 12:37:45 pm
You could try https://forum.opnsense.org/index.php?topic=6590.0 if you haven't seen already.


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: lrosenman on April 05, 2019, 12:39:59 pm
at this point, I'm not running *ANY* IPS (I want to at some point, but I'd like to get the raw performance back to the ~900Meg).
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on April 05, 2019, 12:46:19 pm
Well, the thread here is about netmap which is used in IPS and Sensei only. The IPS performance thread is about IPS performance first and foremost, but offers sysctl tweaks to the driver which are not necessarily specific to IPS.


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: lrosenman on April 05, 2019, 01:19:26 pm
It does help...
see my reply over there.

Download side is still meh, but the upload side is *MUCH* improved.
Title: Re: Call for testing: New netmap enabled kernel
Post by: lrosenman on April 05, 2019, 01:45:22 pm
is there a 19.1.5 version with netmap?
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on April 05, 2019, 02:27:27 pm
Hi franco,

Quote
There's one commit we don't have yet which would suggest a considerable speedup in the IPS department, however:

https://svnweb.freebsd.org/base?view=revision&revision=345269

What remains to be seen is if this requires changes to Suricata to make use of the speedup potential.

Yep, this requires a little work since API is changed for this purpose.
Title: Re: Call for testing: New netmap enabled kernel
Post by: lrosenman on April 09, 2019, 04:41:30 am
any news on 19.1.5 version of a netmap kernel?
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on April 09, 2019, 07:54:10 am
There is no 19.1.5 kernel so there won't be a 19.1.5 netmap kernel. ;)


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: fluxx on May 02, 2019, 09:26:58 pm
Will there be a 19.1.7-netmap kernel?

Greetings,
Christian
Title: Re: Call for testing: New netmap enabled kernel
Post by: franco on May 03, 2019, 09:40:04 am
The kernel only had two patches (one possible kernel panic bug, one possible security issue). I might respin next week, but for now I cannot prioritise.


Cheers,
Franco
Title: Re: Call for testing: New netmap enabled kernel
Post by: Antaris on May 03, 2019, 06:44:23 pm
Thanks anyway, Franco. I also look @ netmap because of vlans and Sensei.
Title: Re: Call for testing: New netmap enabled kernel
Post by: spetrillo on June 02, 2019, 12:18:54 am
So since there was no new kernel in 19.1.7 and 19.1.8 that means I need to install the new netmap via CLI correct? The netmap in plugins is still old?
Title: Re: Call for testing: New netmap enabled kernel
Post by: mb on June 02, 2019, 04:35:55 pm
Hi @spetrillo,

If you're not using virtio ethernet, you do not need to use new netmap kernel for now.

New netmap code is intended to make the netmap infrastructure more stable & robust. Though, there are a few issues that need to be addressed with the new kernel.

A new work is being planned for this. Will keep the thread posted.
Title: Re: Call for testing: New netmap enabled kernel
Post by: ruffy91 on June 03, 2019, 05:19:38 am
The new netmap kernel also made it possible to use Suricata on an interface with multiple VLANs in Promiscous Mode in IPS mode.
Without the netmap kernel all traffic stops as soon as suricata starts.

At least I tought it was the kernel.

Gesendet von meinem MI 9 mit Tapatalk

Title: Re: Call for testing: New netmap enabled kernel
Post by: Oliver on July 08, 2019, 04:22:38 pm
Tried Suricata in this configuration with stable/netmap kernels:
Suricata log with kernel FreeBSD 11.2-RELEASE-p10-HBSD  5e5adf26fc3(stable/19.1)  amd64:
Code: [Select]
Jul 5 18:55:19  suricata: [100282] <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#01-re0" failed to initialize: flags 0145
Jul 5 18:55:19 suricata: [101131] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [101130] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:53:39 suricata: [100282] <Warning> -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml.
Jul 5 18:53:39 suricata: [100102] <Notice> -- This is Suricata version 4.1.4 RELEASE

Suricata log with kernel FreeBSD 11.2-RELEASE-p9-HBSD  4ea457eb7b8(master)  amd64
Code: [Select]
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#01-re0" failed to initialize: flags 0145
Jul 5 19:03:11 suricata: [100215] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100214] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:01:32 suricata: [100103] <Warning> -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml.
Jul 5 19:01:31 suricata: [100168] <Notice> -- This is Suricata version 4.1.4 RELEASE
So Suricata did not start up in either configuration.

Is Suricata expected to work in IPS mode in any of these configurations with the new kernel? Anything else I could try to improve the situation?
Title: Re: Call for testing: New netmap enabled kernel
Post by: Oliver on July 10, 2019, 07:28:05 pm
Diving a bit deeper, this is what is seems like:
I thought I'd put this here for reference.