OPNsense Forum
Archive => 19.1 Legacy Series => Topic started by: mb on February 06, 2019, 12:21:44 am
-
Dear OPNsense community,
One of the exciting new features, introduced with OPNsense 19.1 release, is the introduction of an alternate test kernel having the latest upstream netmap code.
Netmap is a very important subsystem in the base OS, since it provides the necessary plumbing for the operation of Suricata in IPS mode and also Sensei in particular.
Netmap code in FreeBSD (thus HardenedBSD) was almost 4 years old. It lacked lots of new developments and bug-fixes that have been done in this timeframe.
On the FreeBSD side, we (Sunny Valley Networks) sponsored a development effort to bring the latest upstream netmap code into FreeBSD.
Quite promptly, OPNsense team has now landed all this development to OPNsense 19.1. New functionality can be enabled by switching to the new-netmap-kernel (Instructions below)
As said, new kernel brings lots of bug-fixes and new developments, two of the most notable ones are being:
1 - VirtIO network adapters support:
You can now run Suricata/Sensei on virtio adapters. Virtio adapters are found mostly on QEMU/KVM based
Hypervisors like Proxmox, and on Cloud VPS providers.
2 - VLAN child interfaces: You can now run Suricata / Sensei on child vlan interfaces.
There is some more development pending (i.e. native VMware vmxnet support) but as of now and as far as our tests are concerned, we now seem to have a stable netmap implementation.
Bottomline, you should have a more stable Suricata (IPS mode) and Sensei experience after you switch to the new kernel.
Here are the steps for you to run and test the new kernel. Please feel free to share any issues you encountered and we'll do our best to investigate and try to find a solution.
IMPORTANT: Make sure you've completed your upgrade to 19.1. The new kernel is available & compatible with OPNsense 19.1.
To switch to the new-netmap-enabled kernel:
# opnsense-update -bkr 19.1-netmap
After the update & reboot, your 'uname -a' output should be similar: (pay attention to the commit hash and branch, it should be: c4ec367c3d9(master) )
root@fw:~ # uname -a
FreeBSD fw.local 11.2-RELEASE-p8-HBSD FreeBSD 11.2-RELEASE-p8-HBSD c4ec367c3d9(master) amd64
To revert back to the 19.1-default kernel:
# opnsense-update -bkf
Kudos to the OPNsense team for all of their co-operation and help on this.
-
Once I switch my home network over to a new APU4 next month, I'll give this a shot. Were you looking for a specific time frame in which to conclude the testing?
-
@lattera, thx, that'd be great. As for the deadline, it'd be cool if we could see the initial status.. maybe in a couple of weeks?
-
Sounds good. In that case, next chance I get to do network maintenance on the HardenedBSD build infrastructure, I'll test this out.
-
Installed the netmap enabled kernel, seems like it crashes elasticsearch in Sensei constantly. Though, the Sensei service itself is running perfectly. Could it be the case that Sensei is not adjusted yet? Seems I can't activate Sensei on the WAN port - which is a VLAN interface (my provider requires it).
EDIT: rebooted once more (second reboot after kernel installation) and now it seems to work as solid as before.
-
Tested on a VPS driven by OpenStack where it was reproduceable when activating IPS mode all connections were lost with log:
Feb 7 08:57:55 PB-FW1-KARL kernel: 275.156421 [ 401] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb 7 08:57:55 PB-FW1-KARL kernel: 275.156471 [ 720] netmap_update_config configuration changed (but fine)
Feb 7 08:57:55 PB-FW1-KARL kernel: 275.161603 [ 401] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb 7 08:57:55 PB-FW1-KARL kernel: 275.161662 [ 401] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb 7 08:57:55 PB-FW1-KARL kernel: 275.345680 [1026] netmap_finalize_obj_allocator Unable to create cluster at 100380 for 'netmap_buf' allocator
Feb 7 08:57:56 PB-FW1-KARL kernel: 276.048248 [ 79] vtnet_netmap_free_bufs freed 257 mbufs, 0 netmap bufs on 1 queues
Feb 7 08:57:56 PB-FW1-KARL kernel: 276.238981 [ 401] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
Feb 7 08:57:56 PB-FW1-KARL kernel: 276.244971 [ 401] vtnet_netmap_config vtnet config txq=1, txd=512 rxq=1, rxd=512
Updated to netmap kernel and now it runs fine, messages above are gone.
Only message I get is when restarting Suricata:
Feb 7 09:19:00 PB-FW1-KARL kernel: 540.160522 [ 83] vtnet_free_used 1 sgs dequeued from TX-0 (netmap=0)
But machine stays responsive!
Do you also need tests from machines which were not affected to check general functionality?
-
Installed the netmap enabled kernel, seems like it crashes elasticsearch in Sensei constantly. Though, the Sensei service itself is running perfectly. Could it be the case that Sensei is not adjusted yet? Seems I can't activate Sensei on the WAN port - which is a VLAN interface (my provider requires it).
EDIT: rebooted once more (second reboot after kernel installation) and now it seems to work as solid as before.
Good to hear that @jjanzz, any chances you retained some logs regarding Elastic search issue? Might be a good idea to have a look at them. Normally it shouldn't affect ES.
With regard to Sensei, the only difference is that Sensei will be able to run on VLAN and virtio interfaces.
0.7 intentionally refuses to run on those interfaces, because with old kernel it would just cause traffic flow to cease.
One other note regarding WAN interfaces: Sensei is designed to run on inner-looking interfaces. This is because this way, we can also do a mapping between userid and local ip. With WAN interfaces we lose this information (because we get packets after they're NAT'd).
Working on Sensei 0.8-beta1, which should arrive soon. This has virtio/VLAN enabled. So that you can fully enjoy the new functionality with the new kernel.
With Suricata, you can just start testing the new kernel on virtio / VLAN interfaces.
To our experience, virtio on QEMU/KVM gives better performance results compared to em.
-
Updated to netmap kernel and now it runs fine, messages above are gone.
Only message I get is when restarting Suricata:
Feb 7 09:19:00 PB-FW1-KARL kernel: 540.160522 [ 83] vtnet_free_used 1 sgs dequeued from TX-0 (netmap=0)
But machine stays responsive!
Do you also need tests from machines which were not affected to check general functionality?
@mimugmail, very cool.
That log shouldn't indicate a problem. It's netmap telling that it freed a previously used segment.
Our own tests mostly focused on the new functionality. Actually it would be more helpful if you could also test if any existing functionality got broken.
-
Upgraded to the new kernel, rebooted and was met with errors. Glad to provide any pertinent information.
-
@redfish, thanks. How much memory do you allocate to the VM? Does increasing RAM solve?
-
Bare metal with 4GB memory, core i3, and i211 nics. Apologies for the lack of info in previous post.
-
@redfish, thx for the info.
Do you run Suricata on a single interface or on multiple interfaces? Does the old kernel exhibit the same behavior?
Can you paste the output of this command on both kernels:
# sysctl dev.netmap
-
mb, thanks for your inquiries and apologies for the delayed response. As of now, suricata (set to promiscuous mode) is on a single interface (Lan) which has two vlans. It will be a day or so before I’ll have the chance to get the rest of your requested information.
-
redfish, thx, please take your time. I see there are not many interfaces. Let's have a look at what dev.netmap sysctl says.
-
Current system sysctl dev.netmap output
dev.netmap.ixl_rx_miss_bufs: 0
dev.netmap.ixl_rx_miss: 0
dev.netmap.iflib_rx_miss_bufs: 0
dev.netmap.iflib_rx_miss: 0
dev.netmap.iflib_crcstrip: 1
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size_ 2048
dev.netmap.priv_curr_num: 163840
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 4048
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 200
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 73728
dev.netmap.ring_size: 73728
dev.netmap.priv_if_num: 1
dev.netmap.priv_if_size: 1024
dev.netmap.if_cirr_num: 100
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 1024
dev.netmap.if_size: 1024
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 100000
dev.netmap.admode: 0
dev.netmap.fwd: 0
dev.netmap.flags: 0
dev.netmap.adaptive_io: 0
dev.netmap.txsync_retry: 2
dev.netmap.no_pendintr: 1
dev.netmap.mitigate: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0
dev.netmap.ix_rx_miss_bufs: 0
dev.netmap.ix_rx_miss: 0
devlnetmap.ix_crcstrip: 0
-
Thanks to Murat and all of Sunny Valley for providing this rework and helping with a related Suricata incompatibility!
So far this is exciting *and* promising with good overall stability. :)
Ideally, this would be merged into 19.1.x in March or April, although the core team has not yet decided on a timeline.
Cheers,
Franco
-
Installation of the test world+kernel sets on the APU4c4 securing HardenedBSD's build network and infrastructure succeeded without issue. I've got Suricata running in IPS mode and only inspecting traffic on the LAN interface. Seems to be working fine for me for now. I'll be placing the firewall under load over the next few days and will report back.
-
Deployed successfully on another APU4c4 in a lab environment with tunneled IPv6. Working great!
-
Does 19.1-netmap kernel revert anything from 19.1.1 and how will it be kept on par with the upcoming 19.1.2+ versions until it is merged into mainline ?
Just trying to assess the security risks.
-
Also noticed this swap message after reboot:
-
@redfish, this looks like something that needs to be investigated on the Suricata side. Will let you know once I have some more time for this.
@newsense, the only difference between 19.1 default kernel and 19.1-netmap kernel should be the netmap subsystem.
Any chances that you boot with the default kernel and try to see if it produces the same message?
@franco, that's great to hear it'll land on the production branch soon.
@lattera, thanks for the tests. Glad to hear that you had no issues.
-
One thing I did notice was that suricata seems to go crazy when in IPS mode and configured to monitor gif interfaces. Brings the entire network stack down. I'm gonna guess that netmap doesn't work with tunneling interfaces (at least, not yet?) Thank goodness for the serial console port on the APU devices. :)
-
Thanks mb, here are the outputs for both kernels:
Edit: also need to note that I’ve followed this guide https://forum.opnsense.org/index.php?topic=6590.0 and disabled VLAN_HWTAGGING in order for suricata to function with my vlans.
Stock Kernel
dev.netmap.ixl_rx_miss_bufs: 0
dev.netmap.ixl_rx_miss: 0
dev.netmap.iflib_rx_miss_bufs: 0
dev.netmap.iflib_rx_miss: 0
dev.netmap.iflib_crcstrip: 1
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size: 2048
dev.netmap.buf_curr_num: 163840
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 2048
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 200
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 73728
dev.netmap.ring_size: 73728
dev.netmap.priv_if_num: 1
dev.netmap.priv_if_size: 1024
dev.netmap.if_curr_num: 100
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 1024
dev.netmap.if_size: 1024
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 100000
dev.netmap.admode: 0
dev.netmap.fwd: 0
dev.netmap.flags: 0
dev.netmap.adaptive_io: 0
dev.netmap.txsync_retry: 2
dev.netmap.no_pendintr: 1
dev.netmap.mitigate: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0
dev.netmap.ix_rx_miss_bufs: 0
dev.netmap.ix_rx_miss: 0
dev.netmap.ix_crcstrip: 0
New netmap kernel
dev.netmap.ixl_rx_miss_bufs: 0
dev.netmap.ixl_rx_miss: 0
dev.netmap.iflib_rx_miss_bufs: 0
dev.netmap.iflib_rx_miss: 0
dev.netmap.iflib_crcstrip: 1
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size: 2048
dev.netmap.buf_curr_num: 0
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 0
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 0
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 0
dev.netmap.ring_size: 36864
dev.netmap.priv_if_num: 2
dev.netmap.priv_if_size: 1024
dev.netmap.if_curr_num: 0
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 0
dev.netmap.if_size: 1024
dev.netmap.ptnet_vnet_hdr: 1
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 100000
dev.netmap.generic_hwcsum: 0
dev.netmap.admode: 0
dev.netmap.fwd: 0
dev.netmap.txsync_retry: 2
dev.netmap.mitigate: 1
dev.netmap.no_pendintr: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0
dev.netmap.ix_rx_miss_bufs: 0
dev.netmap.ix_rx_miss: 0
dev.netmap.ix_crcstrip: 0
-
@redfish, thanks. Looks like it's on the Suricata side. I'll have a closer look.
-
This is "Excellent" news! ;D. I run a lot of VM with VirtIO on Proxmox VE and have not been able to take advantage of Suricata. Will start converting and testing my OPNSense kernels.
Was very happy with my transition from pfSense to OPNSense, this makes it even better.
Thank you.
-
TheGrandWazoo, great that you found this useful :) Please share your experience with Suricata + new kernel.
-
Now when there is 19.1.1 can we use it with netmap enabled kernel and what is the command?
May be "# opnsense-update -bkr 19.1.1-netmap" ?
-
Antaris, yes, correct. From the first post in the thread:
To switch to the new-netmap-enabled kernel:
# opnsense-update -bkr 19.1-netmap
After the update & reboot, your 'uname -a' output should be similar: (pay attention to the commit hash and branch, it should be: c4ec367c3d9(master) )
root@fw:~ # uname -a
FreeBSD fw.local 11.2-RELEASE-p8-HBSD FreeBSD 11.2-RELEASE-p8-HBSD c4ec367c3d9(master) amd64
To revert back to the 19.1-default kernel:
# opnsense-update -bkf
Franco plans the merge in the upcoming OPNsense 19.1.x minor releases.
-
So is there 19.1.1 netmap enabled or just 19.1 ?
-
There is no base/kernel for 19.1.1 so there is no / won't be a 19.1.1-netmap.
And when 19.1.2 is out with a new base/kernel, then the netmap kernel will be automatically removed as it is a test kernel, but you can lock it from the GUI if you do not want that.
Cheers,
Franco
-
So far 3 days on the new netmap kernel and I haven't had an issue. 3 interfaces two of them are vlans on i350s. Before this I was having a lockup that killed connections to the interfaces that sensei was monitoring that required a complete reboot to get connectivity back.
-
donatom3, that's great to hear. Thanks for sharing your results.
-
Any interest in reports on doing this on AWS EC2? I have a few self-bootstrapped OPNSense setups there that work fine. I can just snapshot and try the kernel to see how that turns out.
-
oneplane, definitely. Thanks for raising this.
We want both OPNsense + Suricata and Sensei to be able to work on all cloud providers including AWS. What's your experience up until now?
-
mb, reverted to stock kernel and suricata is working as intended. Please let me know if there’s anything I can do to help with your investigations.
-
@redfish, any chances doing a few more tests for me? This might isolate the situation a bit more. You can send the results through a private message.
First download nmbridge application.
# fetch https://updates.sunnyvalley.io/nmbridge/nmbridge
# chmod +x nmbridge
This is a very minimal application which does bridging. It's based on netmap. I'd like to see if this also produces the same result:
With the new kernel, try to see if this command produces the same message. Replace ix0 to suit your interface naming. Do this also for the other network interfaces installed on the host.
# ./nmbridge -i netmap:ix0 -i netmap:ix0^
When you run the application make sure to see packets are flowing and your network connectivity is intact for that specific interface.
-
Good day,
I have had NO issues with Suricata IPS in a HA environment using VIRTIO interfaces. I have been running HAProxy environments, SBC (VoIP) environments (A plugin I am working on) and regular firewall environments. Thank you very much!!!!
With that being said 19.1.2 is now out and it has an "OS" update associated with it. Does this 19.1.2 update include this patch set? If not, is there a new kernel that we need to perform the 'opnsense-update'?
Thanks again,
Kev
-
Hi Kev,
There is no refreshed netmap rework kernel yet. If you want to keep it lock the kernel package from system: firmware: packages tab and perform the update as usual.
Look for the release notes, we'll announce the netmap rework there...
Cheers,
Franco
-
Hi Kev,
There is no refreshed netmap rework kernel yet. If you want to keep it lock the kernel package from system: firmware: packages tab and perform the update as usual.
Look for the release notes, we'll announce the netmap rework there...
Cheers,
Franco
Franco,
Thanks for the info. That is what I thought I should do but never hurts to ask. Everything updated without issues.
Thanks again,
Kev
-
There's a new netmap test kernel out there:
# uname -a
FreeBSD sensey.homeoffice.local 11.2-RELEASE-p9-HBSD FreeBSD 11.2-RELEASE-p9-HBSD 4ea457eb7b8(master) amd64
To upgrade use:
# opnsense-update -fbkr 19.1.4-netmap
# opnsense-shell reboot
To revert back to the 19.1.4 default kernel:
# opnsense-update -bkf
The old test sets for 19.1-netmap will be removed from the mirrors now. Changes in the new version include:
o netmap: fix crash with monitors and VALE ports
o ixl: remove unnecessary limitations related to netmap
o netmap: small cleanup on em, lem, igb, ixgbe
o netmap: fix knote() argument to match the mutex state
o netmap: improvements to the netmap kloop (CSB mode)
o netmap: add notifications on kloop stop
o netmap: upgrade sync-kloop support
o netmap: refactor logging macros and pipes
o vmx(4): add native netmap support
o netmap: fix lock order reversal related to kqueue usage
o netmap: remove redundant call to nm_set_native_flags()
o all fixes of the 19.1.2 and 19.1.4 operating system updates
Cheers,
Franco
-
After installing the new netmap-kernel, the Intel network card was no longer found. I ran OPNSense under KVM and passed 2 ports of a hardware Intel card into the VM. The third E1000 is a KVM emulated card. With the new kernel only the virtual card is found. Is there a solution for this ?
Standard kernel: pciconf -l
em0@pci0:3:0:0: class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
em1@pci0:4:0:0: class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
em2@pci0:8:1:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
netmap kernel:
em0@pci0:8:1:0: class=0x020000 card=0x11001af4 chip=0x100e8086 rev=0x03 hdr=0x00
Thank you. I'll take care of it,
Sasha
-
Hi Sasha,
This is very odd as the patches are not supposed to do this.
Is this the case with a normal 19.1.4 as well?
Cheers,
Franco
-
Hello,
In 19.1.4 with the default kernel everything is OK
-
Strange, I went through the commits and saw nothing out of the ordinary. Compared to 19.1-netmap kernel was it the same or working there? I know this might sound strange but what happens if you reinstall 19.1.4-netmap?
Cheers,
Franco
-
I've done that before, but it's always been the same result.
I don't have much experience with FreeBSD, I usually only use Linux and Windows. I would have liked to use the VirtIO network driver under KVM Q35. This works under the standard kernel only with the i440FX chipset from KVM. If anyone has an idea, I would like to try it out.
-
With the new net map kernel sense pi doesn’t detect any traffic on my interface that isn’t a vlan, but it has no issues with the two interfaces that are vlans. This is in intel i211 nics.
I am also not getting any hits in suricata on that non vlan interface.
-
Hi donatom3,
Is the non-vlan interface the trunk (parent) interface for the other vlan child interfaces?
-
MB,
No the vlans were moved to their own interface.
-
Ok, let's have a look at this together.
-
This kernel has also support for native netmap support for vmx(4), VMware VMXNET3 Virtual Interface Controller device.
https://svnweb.freebsd.org/base?view=revision&revision=344272
Native netmap support should yield better performance compared to the emulated driver.
Much appreciated if someone with an existing VMware deployment could test & provide feedback.
PS: Please note that you'll need to set vmxnet3.netmap_native tunable to 1 (from System: Settings: Tunables) to enable native netmap mode.
-
Do setting vmxnet3.netmap_native require reboot to take effect?
-
Hmm, where are we on this now?
We have conflicting reports on this second netmap kernel and the original merge window is approaching now (April/May) so I see potential for delay...
Cheers,
Franco
-
Do setting vmxnet3.netmap_native require reboot to take effect?
Hi Antaris,
Yes, you'll need a reboot. Make sure it's in effect. Use this command:
sysctl vmxnet3.netmap_native
-
Hmm, where are we on this now?
We have conflicting reports on this second netmap kernel and the original merge window is approaching now (April/May) so I see potential for delay...
Cheers,
Franco
Hi Franco,
Agreed. We need to check a few issues reported if they are to be addressed or not.
-
Do setting vmxnet3.netmap_native require reboot to take effect?
Hi Antaris,
Yes, you'll need a reboot. Make sure it's in effect. Use this command:
sysctl vmxnet3.netmap_native
And the result:
sysctl: unknown oid 'vmxnet3.netmap_native'
I added "vmxnet3.netmap_native" in System>>Settings>>Tunables and set it to "1"
-
I added "vmxnet3.netmap_native" in System>>Settings>>Tunables and set it to "1"
Antaris, by checking your configuration, looks like you you're using native netmap for vmx now.
Let me see why tunable does not show up.
-
I'm getting:
root@home-fw:~ # opnsense-update -bkr 19.1-netmap
Fetching base-19.1-netmap-amd64.txz: .. failed, no signature found
root@home-fw:~ # uname -a
FreeBSD home-fw.lerctr.org 11.2-RELEASE-p9-HBSD FreeBSD 11.2-RELEASE-p9-HBSD f083bc4f8a0(stable/19.1) amd64
root@home-fw:~ #
[code]
Is this now broken?
-
Hi Irosenman,
You should now use:
opnsense-update -fbkr 19.1.4-netmap
See: https://forum.opnsense.org/index.php?topic=11477.msg55261#msg55261
-
should this help on em(4) devices with normal speed test type stuff?
I'm on a protectli (https://www.protectli.com) FW1
on ATT Fiber (using https://github.com/aus/pfatt bypass for the RG).
My speeds are ~600meg/sec, but with the Ubiquiti USG I was getting 900+.
Ideas welcome...
I didn't make any changes to the config other than adding the new netmap kernel.
-
I don't expect radical changes on em drivers. There's one commit we don't have yet which would suggest a considerable speedup in the IPS department, however:
https://svnweb.freebsd.org/base?view=revision&revision=345269
What remains to be seen is if this requires changes to Suricata to make use of the speedup potential.
Cheers,
Franco
-
Should I take my performance issue to another topic?
-
You could try https://forum.opnsense.org/index.php?topic=6590.0 if you haven't seen already.
Cheers,
Franco
-
at this point, I'm not running *ANY* IPS (I want to at some point, but I'd like to get the raw performance back to the ~900Meg).
-
Well, the thread here is about netmap which is used in IPS and Sensei only. The IPS performance thread is about IPS performance first and foremost, but offers sysctl tweaks to the driver which are not necessarily specific to IPS.
Cheers,
Franco
-
It does help...
see my reply over there.
Download side is still meh, but the upload side is *MUCH* improved.
-
is there a 19.1.5 version with netmap?
-
Hi franco,
There's one commit we don't have yet which would suggest a considerable speedup in the IPS department, however:
https://svnweb.freebsd.org/base?view=revision&revision=345269
What remains to be seen is if this requires changes to Suricata to make use of the speedup potential.
Yep, this requires a little work since API is changed for this purpose.
-
any news on 19.1.5 version of a netmap kernel?
-
There is no 19.1.5 kernel so there won't be a 19.1.5 netmap kernel. ;)
Cheers,
Franco
-
Will there be a 19.1.7-netmap kernel?
Greetings,
Christian
-
The kernel only had two patches (one possible kernel panic bug, one possible security issue). I might respin next week, but for now I cannot prioritise.
Cheers,
Franco
-
Thanks anyway, Franco. I also look @ netmap because of vlans and Sensei.
-
So since there was no new kernel in 19.1.7 and 19.1.8 that means I need to install the new netmap via CLI correct? The netmap in plugins is still old?
-
Hi @spetrillo,
If you're not using virtio ethernet, you do not need to use new netmap kernel for now.
New netmap code is intended to make the netmap infrastructure more stable & robust. Though, there are a few issues that need to be addressed with the new kernel.
A new work is being planned for this. Will keep the thread posted.
-
The new netmap kernel also made it possible to use Suricata on an interface with multiple VLANs in Promiscous Mode in IPS mode.
Without the netmap kernel all traffic stops as soon as suricata starts.
At least I tought it was the kernel.
Gesendet von meinem MI 9 mit Tapatalk
-
Tried Suricata in this configuration with stable/netmap kernels:
- OPNsense 19.1.9-amd64 on a ZBOX PRO CI329 nano (re0, re1: Realtek PCIe GBE)
- Suricata enabled, IPS mode enabled, Promiscuous mode disabled, Interfaces: LAN
- Hardware CRC, TSO, LRO disabled: all checked (disabling hardware offloading)
- sysctl dev.netmap.admode=1 (otherwise Suricata would block all traffic cf. https://redmine.openinfosecfoundation.org/issues/1688)
- There are VLANS configured on LAN (re0), but these have not been included in Suricata's interface list.
- Other options left at their defaults.
Suricata log with kernel FreeBSD 11.2-RELEASE-p10-HBSD 5e5adf26fc3(stable/19.1) amd64:
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#01-re0" failed to initialize: flags 0145
Jul 5 18:55:19 suricata: [101131] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [101130] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:55:19 suricata: [100282] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 18:53:39 suricata: [100282] <Warning> -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml.
Jul 5 18:53:39 suricata: [100102] <Notice> -- This is Suricata version 4.1.4 RELEASE
Suricata log with kernel FreeBSD 11.2-RELEASE-p9-HBSD 4ea457eb7b8(master) amd64
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - Engine initialization failed, aborting...
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#01-re0" failed to initialize: flags 0145
Jul 5 19:03:11 suricata: [100215] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100214] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:03:11 suricata: [100103] <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for re0, error Operation not supported
Jul 5 19:01:32 suricata: [100103] <Warning> -- [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml.
Jul 5 19:01:31 suricata: [100168] <Notice> -- This is Suricata version 4.1.4 RELEASE
So Suricata did not start up in either configuration.
Is Suricata expected to work in IPS mode in any of these configurations with the new kernel? Anything else I could try to improve the situation?
-
Diving a bit deeper, this is what is seems like:
- Current OPNsense kernels use drivers from the Realtek website. These are based on an old implementation of the FreeBSD re driver, which does not support netmap.
- When Suricata starts up in IPS mode and tries to use netmap, it fails with the Realtek drivers (while the standard FreeBSD re driver would provide the necessary netmap support).
- Even in non-IPS mode, Suricata would sometimes stop working until restarted. On some occasions, log messages "<Error> -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2" appeared in the log at very high frequency. In these cases, stopping Suricata took quite a while.
- My personal impression from observed stability problems and looking at the code is that the Realtek drivers have serious quality issues and should probably not be used in routers.
I thought I'd put this here for reference.