OPNsense Forum
Archive => 20.7 Legacy Series => Topic started by: mb on September 16, 2020, 06:53:51 pm
-
Dear OPNsense community,
It's my pleasure to announce that OPNsense team has shipped the official netmap test kernel today.
This kernel fixes important stability and reliability issues with regard to vmx(4), vtnet(4), ixl(4), ix(4) and em(4) ethernet drivers.
The kernel also adds long-awaited support for tun(4) and lagg(4) interfaces.
The end benefit of this kernel is that you'll be able to run Sensei or Suricata on the following:
- OpenVPN and other VPNs which use tun(4) interface
- Link Aggregation Groups (lagg)
- QEMU/KVM guests with performant vtnet driver
- VMware guests with vmx driver
- Intel 10 Gbps Ethernet drivers
- Intel 1 Gbps Ethernet (em driver) with VLANs
To deploy the new kernel just run below command and restart your firewall.
# opnsense-update -kr 20.7.3-netmap
Patches which went into this kernel have been under heavy testing by us (Sunny Valley Networks) and by the OPNsense team for a few weeks now.
We'd very much appreciate your further testing and feedback. If no further issues pop up, OPNsense team will be shipping all these functionality with 20.7.4 or later releases.
As Sunny Valley Networks, we'd very much like to thank OPNsense/HardenedBSD team, netmap team (Universita di Pisa) and the FreeBSD team for their awesome collaboration and precious efforts. With their full coordination and co-operation, we are able to provide this today.
-
Good news! Today in Webinar I was asked about vmx driver status. Thanks for your efforts!! :)
-
All welcome @mimugmail. Enjoy ;)
-
# opnsense-update -kr 20.7.2-netmap
Just one question, maybe a silly one, is this reversable in case of problems ?
Thanks
-
Awesome!
-
Just one question, maybe a silly one, is this reversable in case of problems ?
No, actually a good question. Yes, you can:
# opnsense-update -kr 20.7.2
-
# opnsense-update -kr 20.7.2-netmap
Just one question, maybe a silly one, is this reversable in case of problems ?
And can I apply this update directly on 20.1.9 or it's better to wait for 20.7.3?
-
@mb
Is it to be expected that now with "20.7.2-netmap" the update functionality proposes to update to "20.7.2"
-
And can I apply this update directly on 20.1.9 or it's better to wait for 20.7.3?
Nope, this is only for 20.7.x releases.
-
Is it to be expected that now with "20.7.2-netmap" the update functionality proposes to update to "20.7.2"
Yes, that's normal. Because you're not running the standard 20.7.2 kernel.
-
Kernel update worked fine with me.
OPNsense 20.7.2-amd64
FreeBSD 12.1-RELEASE-p9-HBSD
OpenSSL 1.1.1g 21 Apr 2020
To doublecheck I'm providing valid testing results for you (PPPoE WAN / Suricata)
1. Only select interface: In my case LAN (vtnet) and don't select WAN (vtnet vlan 6)
2. Add public IP to home networks.
Correct?
-
Nope, it did not change the download speed in APU2. If anything, it got even lower. I use Sensei 1.6 with September definitions.
-
With the new build, should Interfaces Selection for Protected Interfaces be LAN only or can I add vmx0_VLAN's and ovpns OpenVPN Server interfaces also. Wasn't sure if LAN covered them all or not.
-I do have internal DNS server running on a Domain Controller on the LAN. Not sure if that matters for config.
This is a VMWare ESXi 7 environment if the VMX didn't give it away :)
-
To doublecheck I'm providing valid testing results for you (PPPoE WAN / Suricata)
1. Only select interface: In my case LAN (vtnet) and don't select WAN (vtnet vlan 6)
2. Add public IP to home networks.
Hi @heresjody:
1- Correct. Though we have not touched pppoe+netmap yet. Use Sensei on LAN.
2- Not sure if I understood correctly. Can you elaborate?
-
With the new build, should Interfaces Selection for Protected Interfaces be LAN only or can I add vmx0_VLAN's and ovpns OpenVPN Server interfaces also. Wasn't sure if LAN covered them all or not.
-I do have internal DNS server running on a Domain Controller on the LAN. Not sure if that matters for config.
This is a VMWare ESXi 7 environment if the VMX didn't give it away :)
:)
Yes, do not put Sensei on WAN interface; or any interface that Suricata is also running on.
Otherwise, you can add vmx parent/vlan interfaces. You can also add openvpn interfaces.
If you have an internal DNS, try the new realtime dns mapping feature ;)
-
Nope, it did not change the download speed in APU2. If anything, it got even lower. I use Sensei 1.6 with September definitions.
@almadovaris, APU2 is quite weak. Though, I would not expect BW getting lower. Can you share a few figures?
New netmap kernel + Sensei 1.6 is producing better results for us and for some users we're in touch with. We were able to attain around 1 Gbps sustained throughput even with old Xeon servers (ubench scores around 180K-220K - all igb drivers).
-
I only have a free license in a home learning / testing environment but thanks for the info.
Sent from my SM-N986U using Tapatalk
-
I only have a free license in a home learning / testing environment but thanks for the info.
Register for https://sunnyvalley.cloud and you'll find a surprise ;) You can get a trial license for 7 days now - will be publicly announced next week.
-
I only have a free license in a home learning / testing environment but thanks for the info.
Register for https://sunnyvalley.cloud and you'll find a surprise ;) You can get a trial license for 7 days now - will be publicly announced next week.
Is the trial license the surprise lol
Sent from my SM-N986U using Tapatalk
-
Upgrading on a system with igb drivers gains no improvement or benefit correct?
-
Upgrading on a system with igb drivers gains no improvement or benefit correct?
Hi @XeroX, with regard to reliability,stock 20.7.2 is already fine. With the beta kernel and 1.6, we've been reported of performance improvements. If you're fine with your performance, you might want to wait for the release.
-
Nope, it did not change the download speed in APU2. If anything, it got even lower. I use Sensei 1.6 with September definitions.
@almadovaris, APU2 is quite weak. Though, I would not expect BW getting lower. Can you share a few figures?
New netmap kernel + Sensei 1.6 is producing better results for us and for some users we're in touch with. We were able to attain around 1 Gbps sustained throughput even with old Xeon servers (ubench scores around 180K-220K - all igb drivers).
I am a home user, I won't invest thousand Euros for a router. I have unprotected igb2 (i.e. got it outside of Sensei) and that's where my Usenet download boxes are.
In Opnsense 20.1 I got 180 to 200 Mbps download speed through Sensei.
In Opnsense 20.7 with normal kernel I get 65 to 75 Mbps download speed. With the netmap kernel I get 60 to 65 Mbps.
Of course that's measuring the download speed the same way, with Ookla from my internet provider. If I measure it with Fing for mobile phone I get now (netmap kernel) 120 Mbps to 235 Mbps, depending on the router. Through OpenVPN from a computer behind Sensei the speed is again much higher than without VPN.
I did notice that easpect runs about 92% of one processor core during high speed download.
So, yeah, basically I am content about Sensei since my high speed downloads go outside of it.
-
Thanks for the further information.
I am a home user, I won't invest thousand Euros for a router. I have unprotected igb2 (i.e. got it outside of Sensei) and that's where my Usenet download boxes are.
Fair enough.
As a side note, for home-use; at a few hundred, it's possible to find some other solutions in the market.
My colleague is able to saturate his 1Gbps AT&T fiber (Sensei 1.6 and 20.7.2-netmap kernel) with a cheap box:
https://www.speedtest.net/result/c/b847d8ab-56a3-42bc-983e-fc6e5d89adfe
-
Like I said, I got 235 Mbps with Sensei. I suspect that's the maximum my WiFi allows through my mobile phone (I also have other clients for 5 GHz WiFi).
Between 60 Mbps and 235 Mbps there is a huge difference. I don't know how Fing measures speed, but if it would work the same way for Usenet speed, it would be great.
Ok, got it: TLS Usenet connections have a speed of just under 9 MB/s through Sensei. Unencrypted Usenet connections have a speed around 26 MB/s or 27.5 MB/s.
So, unencrypted Usenet reaches about 220 Mbps.
-
To doublecheck I'm providing valid testing results for you (PPPoE WAN / Suricata)
1. Only select interface: In my case LAN (vtnet) and don't select WAN (vtnet vlan 6)
2. Add public IP to home networks.
Hi @heresjody:
1- Correct. Though we have not touched pppoe+netmap yet. Use Sensei on LAN.
2- Not sure if I understood correctly. Can you elaborate?
I’m using Suricata, not Sensei. In the settings of Suricata you can add your public IP to the home network setting. Since PPPoE and netmap hasn’t been touched yet thi setting is irrelevant. For now at least 😬
-
I have pressed 12 and y without knowing in advance that it will revert my kernel. So, after rebooting APU2, I measured the download speed for unencrypted Usenet through Sensei and it's almost 270 Mbps (instead of 220 Mbps with the netmap kernel).
Found the solution: uncheck "Disable hardware large receive offload" save, and reboot. Then netmap kernel is as fast as stock kernel. I mean for unencrypted Usenet connections I got 275 Mbps out of 300 Mbps nominal Usenet speed. But this does not work for SSL or TLS Usenet connections.
-
whats the uname output of the "fixed" kernel?
I did install all available upgrades and ended up with a bootloop so selected previous kernel, disabled sensei 1.6 for the moment at boot while I figure out what I should be seeing ...
this is on ESX 6.7.0 (Build 9484548)
-
FreeBSD 12.1-RELEASE-p9-HBSD
-
FreeBSD 12.1-RELEASE-p9-HBSD
so this is the fixed, correct kernel?
FreeBSD host.name.xyz 12.1-RELEASE-p9-HBSD FreeBSD 12.1-RELEASE-p9-HBSD 3b652d8ad0e(master) SMP amd64
-
3b652d8ad0e is correct, see https://github.com/opnsense/src/commit/3b652d8ad0e
-
ok, re-installed the kernel again, then sensei and its been stable so far - not sure what it was previously
-
@r4nd0m, thanks for the update.
-
@r4nd0m, thanks for the update.
found the issue ... looks like the package does not supersede the original package and will be pulled as an upgrade if upgrades are available leading to the boot loop
I suggest this gets another version number eg 20.7.2.1 so the upgrade wont be suggested back to the non patched kernel
https://i.imgur.com/nk5WZSg.png
-
Boot loops are fixed for me now however performance went down from 270mbps to 120mbps while using esxi with vmx.
Sensei is disabled, only using ips
Is this known to be an issue?
Sent from iPhone via Tapatalk
-
Hi @nines, we are able to attain around 1.2-1.3 Gbps between two VMware guests. ubench score around 400K.
What is your HW configuration? What does ubench -cs tell?
-
Hi @mb thanks for your fast response (as always!)
its an esxi 6.7 running on i5-5300U
https://ark.intel.com/content/www/de/de/ark/products/84862/intel-nuc-board-nuc5i5mybe.html
20.1 ubench
./ubench -cs
Unix Benchmark Utility v.0.3
Copyright (C) July, 1999 PhysTech, Inc.
Author: Sergei Viznyuk <sv@phystech.com>
http://www.phystech.com/download/ubench.html
FreeBSD 11.2-RELEASE-p20-HBSD FreeBSD 11.2-RELEASE-p20-HBSD 07ef86ce9ca(stable/20.1) amd64
Ubench Single CPU: 420580 (0.40s)
20.7 ubench
./ubench -cs
Unix Benchmark Utility v.0.3
Copyright (C) July, 1999 PhysTech, Inc.
Author: Sergei Viznyuk <sv@phystech.com>
http://www.phystech.com/download/ubench.html
FreeBSD 12.1-RELEASE-p9-HBSD FreeBSD 12.1-RELEASE-p9-HBSD 3b652d8ad0e(master) SMP amd64
Ubench Single CPU: 412479 (0.41s)
any thoughts?
-
Hi @nines, all welcome.
CPU score looks good. What happens if you just run nmbridge ? (Both Sensei & Suricata off)
Steps (Assuming vmx0 is your interface)
fetch https://updates.sunnyvalley.io/nmbridge/nmbridge
chmod 750 nmbridge
ifconfig vmx0 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
./nmbridge -i netmap:vmx0 -i netmap:vmx0^
-
what would be the expected output?
618.451661 main [247] same interface, endpoint 0 goes to host
618.488299 nm_mmap [990] do not mmap, inherit from parent
618.488349 main [269] ------- zerocopy supported
618.488377 main [276] Wait 4 secs for link to come up...
622.499677 main [280] Ready to go, vmx1 0x1/1 <-> vmx1 0x0/1.
//
If I try to do it between vmx0 and vmx1: ./nmbridge -i netmap:vmx0 -i netmap:vmx1 the vm instantly becomes unresponsive and the esx console spams "vmx0 drop mbufs that needs checksum offload"
-
Hi Nines,
Do not bridge vmx0 and vmx1, to simulate Suricata/Sensei behaviour bridge the ethernet and the hos operating system.
chmod 750 nmbridge
ifconfig vmx0 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
./nmbridge -i netmap:vmx0 -i netmap:vmx0^
-
Hi mb,
Did that also, it's the first quote in my above post but there is no more output other than this.
// during run of nmbridge no traffic forwarding seems to work, ssh is still working though.
if I control+c traffic forwarding continues to work:
^C864.502603 main [331] poll timeout [0] ev 1 0 rx 0@42 tx 443, [1] ev 1 0 rx 0@69 tx 511
Sent from iPhone via Tapatalk
-
Did that also, it's the first quote in my above post but there is no more output other than this.
// during run of nmbridge no traffic forwarding seems to work, ssh is still working though.
If you're running nmbridge on vmx1, than you need to disable all offloadings on this interface also:
ifconfig vmx1 -vlanhwtso -vlanhwfilter -vlanhwtag -vlanhwcsum -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
./nmbridge -i netmap:vmx1 -i netmap:vmx1^
Then, run another round of speed test, while nmbridge is running. It is ok that nmbridge won't print much output.
-
thanks for your patience.
its much better but not as good as with 20.1 - now its fluctuating between 130 and 275mbps while 20.1 always peaked at 275 (max isp speed)
-
Hi @nines, no worries at all. My pleasure.
If you can see 275 Mbps now; but not always; I'm inclined to think fluctuations might be related to virtual environment. I'd suggest testing while trying to reduce any side effects imposed by other hosts and/or other services on the firewall.
Reason is, in our test environment, our tests show 1.2 Gbps - 1.3 Gbps between two guests. There's some fluctuation, but it's more like 10% or so.
-
ok, I tested again. Simple http download with disabled suricata/sensei is fast at ~270mbit. If ips is off speeds are max, as soon as its enabled suricata is saturating one core with 100% and speed is capped at 120mbit.
So the reson is definetely something related to ips ...
-
Got it, thanks for the update.
-
So I should better go back to 20.1 as you can't reproduce that behavior on your site right?
ruleset is the same so that shouldn't be an issue
Sent from iPhone via Tapatalk
-
Hi @Nines, correct. Netmap-wise, you look good.
WRT IPS, I'd suspect from the configuration & rulesets if you were able to attain higher speeds on 20.1.
-
did some more tests.
its slow if ips is enabled on "wan"
its fast if ips enabled on "lan"
How can that be? What am I missing here? Shouldnt the rules be processed in either scenario? Where does the difference in performance come from?
just for reference:
WRT IPS, I'd suspect from the configuration & rulesets if you were able to attain higher speeds on 20.1.
dont know how because the configuration and rulesets did NOT change.
-
Installed on test vm.
Performance with vmx interfaces are still bad so I have to revert to 20.1.
-
did some more tests.
its slow if ips is enabled on "wan"
its fast if ips enabled on "lan"
How can that be? What am I missing here? Shouldnt the rules be processed in either scenario? Where does the difference in performance come from?
Hi @Nines, nmbridge test shows the results with regard to the netmap & kernel alone. If this provides the expected result, this means you're good with regard to the kernel & netmap.
For IPS, unfortunately I'm not so much deep into the details. How about discussing it in the IPS forum?
-
Hi mb, as I wrote in #42 ithe results with nm bridge were better but there was still a huge difference compared to 20.1 WITH enabled ips.
In my oppionion the current kernel is definetely not performing as good as with 20.1 and as such the performance with enables ips is even worse.
Yes, ubench result is good, but overall performance is far away from 20.1. I know you and your team have put a huge amount of work into this but I think it's better to point out potential issues then having to deal with the complaints later on.
To be clear: you can definetely assume that the results mentioned here are not sourced from external factors like esx load, webserver load or something else. I can easily check this by just restoring both vmware snapshots back and forth - it MUST have something to do with 20.7 and/or the kernel. The results are strictly "following the problem atic version" and never appear with 20.1 (as said, performance with 20.1 and enabled ips is always a lot better then 20.7 with disabled ips, rest of the config is exactly the same)
Thanks again for your support and work.
Sent from iPhone via Tapatalk
-
Still going strong with the test kernel. On both my em and my vtnet interface servers no crashes since I installed in 7 days ago.
Just checking: Can I safely apply the netmap test kernel to 20.7.3?
-
Kind of a same question, I just updated to v20.7.3 and came from the 20.7.2-netmap kernel. Now my wireguard side2side tunnel does not connect anymore.
Is there something like opnsense-update -kr 20.7.3-netmap so I can proceed testing the new kernel?
-
Hi @heresjody, @andreas, it's best to wait for @franco to announce 20.7.3-nemap kernel. It's most probably not there yet.
-
Upsi, then I was too ambitious....
root@OPNsense:~ # opnsense-update -kr 20.7.3-netmap
Fetching kernel-20.7.3-netmap-amd64.txz: ......... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-20.7.3-netmap-amd64.txz... done
Please reboot.
-
Upsi, then I was too ambitious....
root@OPNsense:~ # opnsense-update -kr 20.7.3-netmap
Fetching kernel-20.7.3-netmap-amd64.txz: ......... done
!!!!!!!!!!!! ATTENTION !!!!!!!!!!!!!!!
! A critical upgrade is in progress. !
! Please do not turn off the system. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Installing kernel-20.7.3-netmap-amd64.txz... done
Please reboot.
😂😂 But does it work? 🤔
-
I guess @andreas will make the 20.7.3-netmap announcement soon ;)
-
Hehe, sorry, that was not my intention, I thought, if no package is available yet, it would not find anyhing and therefor not proceed.
That's why I surprisped myself slightly :P. So as said, I guess too ambigious, it rebooted, but kernel does not look much different (it's from 21.09.).
-
We put 20.7.3-netmap there for users to find to provide all the updates of 20.7.3 with all the netmap things from master branch plus Realtek driver update. No need to wait for approval. ;)
Cheers,
Franco
-
Cool, thanks @Franco, so if I understood you correctly, then I'm back on the netmap testing track with my update now? So the update worked as initially assumed by me?
-
Yep. Last test kernel, I promise.
Cheers,
Franco
-
OPNsense 20.7.3-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1g 21 Apr 2020
AMD Athlon 3000G with Radeon Vega Graphics (4 cores)
8GB RAM and Intel E1G42ET Dual Ethernet
Virgin Gig1 Broadband
When downloading large files via an SSL encrypted connection with Sensei 1.6 in standard Routed Mode and the latest 20.7.3 netmap kernel, my download speed maxes out at about 450mbps, but running Speedtest I get about 930mbps. If I change Sensei to Routed Mode with generic netmap driver, SSL downloads peak at 580mbps. If I switch mode to Passive, then SSL downloads reach max speed about 980mbps.
Doing the same SSL download with 20.1 and Sensei in standard Routed Mode I will max out my broadband speed.
Hoping this info can help improve the netmap kernel.
Thanks for the great work being done!
-
Hi AceRickslick,
Thanks for the detailed information. What does ubench tell about your cpu score:
ubench -cs
Which ethernet adapter are you using for Sensei-protected interfaces and for the WAN interface?
Can you get 930 Mbps with Sensei on during speedtest test?
-
Hi mb,
ubench score is 574396.
For Sensei-protected it is igb0 and igb1 is used for WAN. It's an Intel 82576 Chipset ethernet adapter.
Yes, I can get 930 Mbps with Sensei turned on during a speedtest test. Just can't hit those speeds during SSL downloads where before on 20.1 it worked perfectly.
-
Thanks, that's promising.
Can I ask for one more test: what happens if you put Sensei on bypass mode? This will tell if this is related to Sensei or netmap.
-
When Sensei is in Bypass Mode (Passive Mode), I get full speed on SSL downloads and Speedtest test, about 980 Mbps.
-
Be aware, Passive and Bypass mode are different. Passive mode do not use netmap at all. You can enter Bypass mode via Sensei -> Status -> Enter Bypass Mode
Is it Bypass mode?
-
My bad it was in Passive Mode, I have retested in Bypass Mode and I get full Gigabit speeds on Speedtest and SSL downloads, about 980 Mbps.
Also just to rule out busy network times/time of day, when I ran the above test, I also tested Sensei in the default Routed Mode, Speedtest hits about 980 Mbps but on the exact same SSL file download as above the speed never gets above 480 Mbps.
-
Hi @Ace,
No worries at all, and thanks for the update. Now it's clear. It looks like we need an optimization on SSL-based classifier.
I'll update the thread once we have some results.
-
question: when I try the test kernel with that command - "opnsense-update -kr 20.7.3-netmap" and reboot afterwards, how do I identify with "uname -a" if the "right" kernel (the netmap-test-kernel) is used?
currently I get the following:
FreeBSD OPNsenseTEST.test 12.1-RELEASE-p10-HBSD-FreeBSD 12.1-RELEASE-p10-HBSD #0 517e44a00df(stable/20.7)-dirty: Mon Sep 21 16:21:17 CEST 2020 root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
-
I just upgraded to the latest 20.7.3 release from 20.1.9 now that this should be the last netmap test kernel.
I disabled Suricata and Sensei first and then proceeded with the upgrade. The upgrade had no issues and the 20.7.3-Netmap kernel installed successfully as well. I then re-enabled Suricata and Sensei, however after everything was back up and running I noticed that my download bandwidth has been reduced considerably as others have mentioned in previous posts as well. Below are my results including general configuration details.
Host: ESXi 6.7 Update 3 (Build 16713306)
OPNsense VM: 4 vCPUs and 8GB RAM
Physcial CPU: Intel Core i7-4790S
Interfaces: vmx0 (LAN - Sensei), vmx1 (WAN - Suricata), no VLANs
Max Download Speed: 750mbps
OPNsense 20.7.3 with 20.7.3-Netmap Kernel:
Suricata Disabled and Sensei PE Stopped (Native) - 720mbps
Suricata Disabled and Sensei PE Bypassed (Native) - 600mbps
Suricata Disabled and Sensei PE Enabled (Native) - 320mbps
Suricata Enabled (Hyperscan/High Profile) and Sensei PE Stopped (Native) - 440mbps
Suricata Enabled (Hyperscan/High Profile) and Sensei PE Bypassed (Native) - 600mbps
Suricata Enabled (Hyperscan/High Profile) and Sensei PE Enabled (Native) - 420mbps
Odd that with Suricata Disabled the throughput goes down, and with Suricata Enabled with the Sensei packet engine bypassed the throughput actually goes up when compared to the stopped state.
OPNsense 20.1.9:
Suricata Enabled and Sensei Enabled - 720mbps
Speedtest was run 3 times in each configuration using the same speedtest server.
I also tried Suricata with the Medium/High & Custom Profile's with no effect on throughput, so I left it on High.
I tested the "generic/emulated" netmap driver through the Sensei config, and the throughput on that was atrocious by dropping the download speed to 15mbps. This is the only setting that also affected my upload throughput.
With Sensei 1.6 my Wireguard interface does not show up as an available interface to add as a protected interface, this should now be possible with Sensei 1.6 correct?
I noticed that once Suricata starts it uses 1 CPU core @ 100% for about 5mins (After Boot as well), during this time you can not run a speedtest as it affects throughput by about 100-150mbps in my setup. Not sure this is relevant, but figured I would mention it as this is new from my OPNsense 20.1.9 install.
-
question: when I try the test kernel with that command - "opnsense-update -kr 20.7.3-netmap" and reboot afterwards, how do I identify with "uname -a" if the "right" kernel (the netmap-test-kernel) is used?
Hi @the-mk, mine shows the following
# opnsense-update -kr 20.7.3-netmap
Your system is up to date.
# uname -a
FreeBSD fw.local 12.1-RELEASE-p10-HBSD FreeBSD 12.1-RELEASE-p10-HBSD #1 ebb8c1489c7(master)-dirty: Mon Sep 21 13:50:27 CEST 2020 root@sensey64:/usr/obj/usr/src/amd64.amd64/sys/SMP amd64
-
Hi @xpendable, thank you for the detailed test results.
Can you reach out to us via "Report Bug" menu from the Sensei UI? We'd like to have a closer look at your configuration & system.
PS: Make sure you check all options and share relavent information.
-
Using Netmap, e.g. by turning on Sensei on LAN or Suricata in IPS mode on WAN, the traffic graph on the dashboard stops working for those interfaces, and Zabbix agent is unable to gather interface statistics, too.
-
Hi @athurdent,
Traffic Graph problem existed with the early 20.7 release but it isn't anymore with 20.7.2 and 3.What is your OPNsense version?
-
Hi @athurdent,
Traffic Graph problem existed with the early 20.7 release but it isn't anymore with 20.7.2 and 3.What is your OPNsense version?
Odd, I‘m on OPNsense 20.7.3-amd64 with this experimental kernel, double checked. Happens with igb and vtnet here.
Cannot attach my screenshot from the iPad as it is too big it seems, so you have to take my word for it. 😅
-
I confirm that I do not get a kernel panic when using vmxnet3 in ESXI and IDS or Sensei and the 20.7.3-netmap kernel.
But that's all I can confirm is no kernel panic, it is still very slow. My simple but very trusted/reliable test is to use iperf3 on both the opnsense vm and another local vm on in the same ESXI host.
iperf3 -c 192.168.1.1 -R (using -R to simulate download speed to machine)
With IDS off
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 2.27 GBytes 1.95 Gbits/sec 0
With IDS on
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.06 GBytes 915 Mbits/sec 2375
Look at the number of retransmissions. It's not bound by memory or CPU. CPU never goes over 20% and memory has many GB's left.
E3-1285V6 - 4 CPU's allocated and 12GB ram.
-
Hi @gauthgig, thanks for the figures. Couple of questions:
1. By "slow", do you mean 915Mbps or do you have lower speeds than this?
2. How was the situation with OPNsense 20.1.x?
3. How does Sensei bypass mode behave?
-
By slow I mean a drop from 1.95Gbs to 0.915Gbs, 50% reduction.
In 20.1.x I was seeing about 1.7Gbs, so much less drop when netmap enabled.
I only showed the Suricata on LAN. I'll re-run with sensei normal and bypass more and send the results. By the way, my ELK stack is on another ESXI with a 10Gbs link, so the ELK CPU/Memory load will not impact opnsense/sensei.
-
Thanks, looking forward to Sensei results.
Does anyone know if OPNsense 20.7.3 and 20.1.9 have different Suricata releases, or is it the same major release?
-
@mb as far as I know it's suricata 4x vs suricata 5.x
Sent from iPhone via Tapatalk
-
By slow I mean a drop from 1.95Gbs to 0.915Gbs, 50% reduction.
In 20.1.x I was seeing about 1.7Gbs, so much less drop when netmap enabled.
I only showed the Suricata on LAN. I'll re-run with sensei normal and bypass more and send the results. By the way, my ELK stack is on another ESXI with a 10Gbs link, so the ELK CPU/Memory load will not impact opnsense/sensei.
this is normal for an IDS, it inspects every packet, if you enable all rules this is even optimistic. disabling some rules may show a noticeable performance increase with IDS enabled
-
Good news - I got the 20.7.3-netmap working great with both Suricata (WAN) and Sensei (LANs) working great.
I posted some stats yesterday that showed a 50% drop which never happened in 20.1.x (normally about a 3-5% drop on my hardware). Here are the new results:
Internal Testing - controlled no other network noise
iperf3 -c 192.168.1.1 -R (using -R to simulate download speed to machine)
With Sensei off
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.37 sec 4.04 GBytes 3.34 Gbits/sec 0
With Sensei on
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.37 sec 3.92 GBytes 3.14 Gbits/sec 0
Going in or out from the LAN to the internet (Passing both Suricata and Sensei Netmap interfaces)
Speedtest
Both IDS off
Download: 903.26 Mbit/s Upload: 956.85 Mbit/s
Both IDS on
Download: 891.26 Mbit/s Upload: 910.85 Mbit/s
What Changed - from yesterday - ESXI, recently I did a upgrade from ESXI6.7 to 7.0. I found an article about memory delays which a patch is coming for. In the meantime it was recommended to RESERVE ALL Memory for latency sensitive VMs. I did that and now netmap is working great. Looking forward for the test kernel to be rolled into the news production update along with the ESXI patch. I was even able to drastically reduce the resources available to opnsense, here is what the final test numbers above were on:
E3-1285V6 - 2 cpus, 6G RAM
-
Hi @gauthig,
I tried the reserve all memory option for my OPNsense VM in ESXi 6.7, unfortunately it made no difference for my setup. Currently I can not upgrade to ESXi 7 due to hardware compatibility issues. However my OPNsense upgrade was in place and was the only thing that changed in my environment. So I'm guessing that the performance issues are with OPNsense 20.7 and the new netmap kernel.
-
Just wanted to add in my thanks, working on my end and has fixed my flakey gig connection ranging from 400-600Mbps to sit solid at a gig!
-
I was able to improve my download bandwidth by increasing the network transmit and receive descriptors, I'm now getting around 600mbps. So not as high when I was on OPNsense 20.1.9, but I'm fairly satisfied with my download speed now. Hopefully there are further improvements in the future that will increase performance even more.
Just as a fyi... I did try a number of VMware ESXi settings such as memory reservation, CPU affinity, CPU reservation and setting the latency of the VM in VM options/advanced to High with no improvements.
-
With the Sensei 1.6.1 update I am now getting around 700mbps due to the SSL/TLS performance improvements. During the test it actually reaches speeds into the 750-780mbps but can not sustain it and drops to around 700mbps. So great work in improving the throughput in Sensei 1.6.1.
I noticed that this increase was only noticed in SSL/TLS connections (HTTPS) as that is what they were meant to address. Can these improvements also be made for unencrypted connections? On an HTTP only speed test the results stay around 530mbps for me.
-
Yup, agree, Ookla from my ISP now gives 138 Mbps instead of 60 to 75 Mbps. Ookla measures by SSL connections.
Also, there is no longer a large speed difference between unencrypted and encrypted Usenet downloads.
-
@gauthig, @keanu @ @almodovaris, @xpendable, Thanks for the update and insights.
Glad to hear that TLS/SSL speeds are up. We're also scrutinizing the HTTP processor to see if we can save more cpu cycles.
-
Using Netmap, e.g. by turning on Sensei on LAN or Suricata in IPS mode on WAN, the traffic graph on the dashboard stops working for those interfaces, and Zabbix agent is unable to gather interface statistics, too.
Curious if there has been any progress in resolving this? Feels like it's been broken a long time.
-
Hi @FullyBorked,
It's on our agenda now. It seems to be related to the ordering of bpf(4) processing in iflib(4). One of the first issues we'll have a look following 20.7.4 release.
-
Hi @FullyBorked,
It's on our agenda now. It seems to be related to the ordering of bpf(4) processing in iflib(4). One of the first issues we'll have a look following 20.7.4 release.
Awesome, thanks for the update and the hard work. 8)
-
I updated to 20.7.4, which includes the modified netmap, but I still have the bandwidth problem. If I enable Sensei my connection drops from 850Mb to about 420Mb. I'm using em driver on Vmware 6.7.
-
Hi @RickyTR, can you share the hardware details of the hypervisor? Did you have a chance to try with vmx driver?
-
Hi @mb. The hardware is 2 CPUs x Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz with 8GB RAM. I don't think the problem may be related to hw performance because with Sensei active the Cpu is around 5% and never goes over 60%. I tried with vmx with the same result.
-
Hi RickyTR,
Make sure you track cpu usage via "top -SP" and see if any CPU core is fully utilized.
And one more question: are you testing the speed to the internet or between two local LANs?
-
Hi mb,
top -SP is not showing anything strange, no single CPU is at 100%. I tried testing the speed to the internet.
-
Hi @RickyTR, thanks for more information. Your CPU looks decent. You should be able to saturate a 1 Gig WAN connection.
With a Dual core Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz / OPNsense 20.7.4; I am able to do 930-940Mbps (both Sensei/Suricata on):
https://www.speedtest.net/result/10321829371
Virtualization comes into the scene as a new variable which might change things. Any chances that you can reach out support? Let us run a few tests on your system.
-
Hi @Rickytr,
I had similar problems in ESXi with the vmx adapters. I found that increasing the interfaces transmit and receive descriptors provided a big improvement. You could also try increasing the amount of transmit and receive queues as well.
This option (hw.pci.honor_msi_blacklist="0") is required in order to enable the number of transmit and receive queues to be overridden. Also the number of queues need to be less then or equal to the number of CPU cores assigned to your VM.
These values override the transmit (ntxqs) and receive (nrxqs) queues:
dev.vmx.0.iflib.override_ntxqs="4"
dev.vmx.0.iflib.override_nrxqs="4"
These values override the transmit (ntxds) and receive (nrxds) descriptors:
dev.vmx.0.iflib.override_ntxds="0,2048"
dev.vmx.0.iflib.override_nrxds="0,1024,0"
Since I have 2 vmx interfaces I specify which interface these overrides are applied to:
1st vmx interface (dev.vmx.0) - LAN
2nd vmx interface (dev.vmx.1) - WAN
Create /boot/loader.conf.local if not already created, and add the following and change the ntxqs/nrxqs queue values to match the number of your CPU cores. Also you can adjust the ntxds/nrxds values as needed, but keep in mind that the vmx interface has a maximum value of 4096 for the transmit descriptors and a maximum value of 2048 for the receive descriptors.
# VMware tunables for vmx interfaces
hw.pci.honor_msi_blacklist="0"
dev.vmx.0.iflib.override_ntxqs="4"
dev.vmx.0.iflib.override_nrxqs="4"
dev.vmx.0.iflib.override_ntxds="0,2048"
dev.vmx.0.iflib.override_nrxds="0,1024,0"
dev.vmx.1.iflib.override_ntxqs="4"
dev.vmx.1.iflib.override_nrxqs="4"
dev.vmx.1.iflib.override_ntxds="0,2048"
dev.vmx.1.iflib.override_nrxds="0,1024,0"
Remember to take a snapshot of your VM before making any changes, so you have an easy way to recover just in case.
There is a good explanation of these tunables here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237166
The tunables provided in the link below have no affect, however the information on the queues and descriptors are still valid:
https://www.freebsd.org/cgi/man.cgi?query=vmx&sektion=4
-
I use ESXi 7.0.1 (VMs upgraded to same version) and recently have upgraded from 20.1.9 to 20.7.4. Esxi runs on Supermicro board with Intel NICs (and vmxnet3 in VMs). Sensei is active on vlans parent interface, no Suricata running.
Inside my LAN I'm able to saturate my 1 Gb network:
Connecting to host 172.16.1.1, port 59242
[ 5] local 172.17.0.2 port 36110 connected to 172.16.1.1 port 59242
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 111 MBytes 932 Mbits/sec 15 704 KBytes
[ 5] 1.00-2.00 sec 109 MBytes 913 Mbits/sec 0 816 KBytes
[ 5] 2.00-3.00 sec 109 MBytes 912 Mbits/sec 0 912 KBytes
[ 5] 3.00-4.00 sec 109 MBytes 912 Mbits/sec 0 1000 KBytes
[ 5] 4.00-5.00 sec 110 MBytes 923 Mbits/sec 0 1.06 MBytes
[ 5] 5.00-6.00 sec 106 MBytes 891 Mbits/sec 235 594 KBytes
[ 5] 6.00-7.00 sec 108 MBytes 902 Mbits/sec 31 554 KBytes
[ 5] 7.00-8.00 sec 109 MBytes 912 Mbits/sec 0 690 KBytes
[ 5] 8.00-9.00 sec 109 MBytes 912 Mbits/sec 0 802 KBytes
[ 5] 9.00-10.00 sec 108 MBytes 902 Mbits/sec 0 899 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.06 GBytes 911 Mbits/sec 281 sender
[ 5] 0.00-10.01 sec 1.06 GBytes 908 Mbits/sec receiver
iperf Done.
And never had a problem to almost saturate my internet connection 300 Mb/s up/down.
But still I use tunable "vmxnet3.netmap_native" as per Re: Call for testing: New netmap enabled kernel (https://forum.opnsense.org/index.php?topic=11477.msg55913#msg55913) Shall I remove it?
-
For me nothing helped but switching to e1000 AND change the ips algorithm to the new keen style variant.
Note: at least in my case, that only seems to work with 20.7.4 and nothing before
Gesendet von iPhone mit Tapatalk
-
Hello all,
I am trying to install Zenarmor onto my firewall. I have several ports(Intel I350 NICs) in a LAG and Zenarmor is saying the driver for the interface(LAG?) is incompatible with Netmap. I am trying to run Zenarmor in passive mode for now. How can I resolve this? I am running 22.1.5 of OPNsense.
Thanks,
Steve