Sensei throughput cap despite high-performance device

[mihak]

I am running OPNSense on a dedicated i7 CPU with 32 GB of memory and 6 gbps ports. Ubench CPU 1132791 and Ubench MEM 2337171. My internet connection is 1 gpbs.

Before installing and enabling Sensei, average throughput on fast.com or speedtest.com was close to 1.0 gbps with a usual overhead penalty. But when Sensei is installed and active on LAN ports (L3 mode with either native or generic nmap driver), throughput drops to 250 mbps - mere 25% of available bandwidth. CPU is idling and never goes above 15%.

I installed the new 20.7.3-netmap driver - but that didn't change the throughput at all.

What am I doing wrong? What troubleshooting data would you like to see?

[mr.yx]

i am just curious, what i7 do you have that gets those scores? did u bench it with the singlecore flag (ubench -sc)?

a modern coffeelake with 4.2ghz gets arround 690k on single...

[mb]

Hi @mihak,

We received your report. It looks like 1M cpu score is multi-core score, since single core cpu score seems around 270.000.

Anyhow, I believe you should still be able to attain higher speeds. We'll follow up with you.

[mihak]

SVN team did some analysis on my router today and:

- confirmed that throughput indeed drops from >900 Mbps to ~250 Mbps when Sensei is on and active
- throughput goes back to >900 Mbps when Sensei is in bypass mode
- disabling the hyperthreading of firewall increased the throughput to ~350 Mbps

My device is one of the new-generation Protectli clones: https://www.aliexpress.com/item/4000803229693.html

i7 CPU with 32 GB ram and 500 GB mSATA

Will post an update once we progress more.

[almodovaris]

On my APU2C4 (4 cores AMD at 1000 MHz) I have now 356 Mbps to my own provider, through Sensei.

As told by Gary7, I have applied the following tweaks in /boot/loader.conf.local :

vm.pmap.pti = 0
hw.ibrs_disable = 0

[Anael]

SVN team did some analysis on my router today and:

- confirmed that throughput indeed drops from >900 Mbps to ~250 Mbps when Sensei is on and active
- throughput goes back to >900 Mbps when Sensei is in bypass mode
- disabling the hyperthreading of firewall increased the throughput to ~350 Mbps

My device is one of the new-generation Protectli clones: https://www.aliexpress.com/item/4000803229693.html

i7 CPU with 32 GB ram and 500 GB mSATA

Will post an update once we progress more.

Is that a 7th generation proc ? I have an i5 8th and it seems that i got more physical CPUs. 

[ittk]

SVN team did some analysis on my router today and:

- confirmed that throughput indeed drops from >900 Mbps to ~250 Mbps when Sensei is on and active
- throughput goes back to >900 Mbps when Sensei is in bypass mode
- disabling the hyperthreading of firewall increased the throughput to ~350 Mbps

My device is one of the new-generation Protectli clones: https://www.aliexpress.com/item/4000803229693.html

i7 CPU with 32 GB ram and 500 GB mSATA

Will post an update once we progress more.

Is that a 7th generation proc ? I have an i5 8th and it seems that i got more physical CPUs. 

As far as i know, opnsense and sensei are not capable of using full cpu Power when you have multi-core or Multi threded CPUs. Only Single core with high MHz will get Performance, but not when there are multiple cores. Also Network stack is lacking Features like RSS and PCBGROUP kernel options with supoorting NIC Adapter  and so on to gain full Power out of your modern hardware...

https://forum.opnsense.org/index.php?topic=19420.0

[alexroz]

Do I get it right - according to https://docs.opnsense.org/vendor/sunnyvalley/sensei_hardwarerequirements.html#cpu-memory Sensei cant provide bandwidth above 500 Mbps?

[mb]

Hi @ittk, thanks for the pointer. Yes, Sensei, as of now, will be able utilize a single core "per interface".
If you have multiple interfaces, those will be assigned to seperate worker processes, thus seperate CPU cores.

To our experience so far, a vast majority of the current Sensei user base has a WAN connection either 1Gbps or
lower. So our focus has been to deliver a good experience for this segment of the user base.

Work is ongoing for multi-core support per interface.

@alexroz, that chart is a bit conservative We'll post an update to the relavent documentation based on the latest figures from the field.

For 1Gbps, a modern CPU (i3 or later) with an ubench score higher than 300.000 should be enough to attain 1Gbps.
I have an Intel i5-5300U CPU @ 2.30GHz at home and can do 1Gbps with AT&T uplink.

Those who are in need of multi-gigabit speeds with Sensei; our experience has been that below CPU is one
of those which is able to provide around 3Gbps throughput using a single CPU core:

https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1285+v6+%40+4.10GHz&id=3158

[alexroz]

I am running OPNSense on a dedicated i7 CPU with 32 GB of memory and 6 gbps ports. Ubench CPU 1132791 and Ubench MEM 2337171. My internet connection is 1 gpbs.

Before installing and enabling Sensei, average throughput on fast.com or speedtest.com was close to 1.0 gbps with a usual overhead penalty. But when Sensei is installed and active on LAN ports (L3 mode with either native or generic nmap driver), throughput drops to 250 mbps - mere 25% of available bandwidth. CPU is idling and never goes above 15%.

I installed the new 20.7.3-netmap driver - but that didn't change the throughput at all.

What am I doing wrong? What troubleshooting data would you like to see?

Have anyone tried to apply the following hardware performance optimization technique with opnsense?
https://teklager.se/en/knowledge-base/opnsense-performance-optimization/  ?

[almodovaris]

Yup, I have applied those tweaks. They are APU specific. But it is wise to get inspired from them and see what works for you. Sensei speed is unlikely to change without changing the kernel.

[nferocious76]

It seems it's very demanding. And my current hardware won't be able to keep up.
I guess it's not for me. 
https://docs.opnsense.org/vendor/sunnyvalley/zenarmor_hardwarerequirements.html

[jclendineng]

Following for a fix, I keep having to uninstall as it barely hits gigabit (10gb network here). I see multi gig on the roadmap, maybe I’ll circle back when it doesn’t cut my speed so dramatically.

[beki]

Hi jclendineng,
Did you try the latest OPNsense 23.1.6 and select the emulated netmap mode for Zenarmor deployment?
The latest kernel includes some netmap improvements.
Bests

[jclendineng]

I have not! I’ll give it a shot when I get home next week. Can’t do remote as last few installs have also just hard locked the system when adjusting anything touching netmap