Zenarmor throughput with N100 / i226v

[athurdent]

Just go a HUNSN RJ42 in (shipped from Amazon Germany, https://www.amazon.de/dp/B0C985FVT1 ).
Installed Proxmox and passed through two NICs to an OPNsense VM.
Without Zenarmor, full 2.5G throughput, measured through the box with a local 10G iperf3 server on my WAN.
With Zenarmor Free edition (NICs are in L3 with native netmap driver, seems to work fine) it looks like this

Code: [Select]

iperf3 -R -t60

[  5]   0.00-60.04  sec  14.7 GBytes  2.10 Gbits/sec  1957             sender
[  5]   0.00-60.00  sec  14.7 GBytes  2.10 Gbits/sec                  receiver

iper3 -t60

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  10.3 GBytes  1.48 Gbits/sec  3800             sender
[  5]   0.00-60.04  sec  10.3 GBytes  1.48 Gbits/sec                  receiver
Awesome! 

[sy]

Hi @athurdent,

What about with emulated netmap driver?

[athurdent]

Hi @sy,

looks like this with the emulated driver

Code: [Select]

iperf3 -R -t60

[  5]   0.00-60.04  sec  8.93 GBytes  1.28 Gbits/sec  1245             sender
[  5]   0.00-60.00  sec  8.93 GBytes  1.28 Gbits/sec                  receiver

iper3 -t60

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  10.8 GBytes  1.55 Gbits/sec  2117             sender
[  5]   0.00-60.04  sec  10.8 GBytes  1.54 Gbits/sec                  receiver

[athurdent]

And here's the blind test, no Zenarmor, forgot it in the OP.

Code: [Select]

iperf3 -R -t60

[  5]   0.00-60.04  sec  16.4 GBytes  2.35 Gbits/sec  842             sender
[  5]   0.00-60.00  sec  16.4 GBytes  2.35 Gbits/sec                  receiver

iper3 -t60

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  16.4 GBytes  2.35 Gbits/sec  1776             sender
[  5]   0.00-60.04  sec  16.4 GBytes  2.35 Gbits/sec                  receiver

[athurdent]

Here's a fresh test with a MacBook Pro using a 2.5G adapter, this time with 5 concurrent streams.
I'm getting line rate throughput, fantastic!

Code: [Select]

% iperf3-darwin -c192.168.178.8 -R -P5 -t60
...
[SUM]   0.00-60.00  sec  16.1 GBytes  2.30 Gbits/sec  22304             sender
[SUM]   0.00-60.00  sec  16.0 GBytes  2.30 Gbits/sec                  receiver


Code: [Select]

% iperf3-darwin -c192.168.178.8 -P5 -t60
...
[SUM]   0.00-60.00  sec  16.2 GBytes  2.32 Gbits/sec  5542680             sender
[SUM]   0.00-60.01  sec  16.2 GBytes  2.32 Gbits/sec                  receiver


[gregg098]

Any chance you could test with the same config, but virtial NICs (Virtio - default options) in the OPNsense VM vs passthrough? One with and one without Zenarmor would be awesome. I'm getting a similar machine soon and have always run OPNsense with virtualized NICs.  Thanks.

[almodovaris]

An important update (multicore eastpect) is expected in October. So, wait with your measurements until that update is published.

[Keaton Mertz]

This reduction in throughput can be attributed to various factors related to the configuration and performance of Zenarmor (OPNsenseGeometry Dash) within your Proxmox virtualized environment.

[athurdent]

This reduction in throughput can be attributed to various factors related to the configuration and performance of Zenarmor (OPNsenseGeometry Dash) within your Proxmox virtualized environment.

I would not call line rate throughput a reduction… 😉

[extraomelette]

This reduction in throughput can be attributed to various factors related to the configuration and performance of Zenarmor (OPNsenseGeometry Dash) within your Proxmox virtualized environment.

run 3

I also agree with this point of view. Many answers to the problem are being raised. I'm doing a lot of research and testing.

[dss]

What's the best way to set up iperf3 for testing OPNsense/Zenarmour throughput? I've got a CWWK N100 / i226v firewall and if I run iperf3 as a server on the firewall and plug a macbook pro with a 2.5gbe into the LAN port, I'm not getting close to 2.5Gbe speeds when Zenarmour is enabled. But perhaps running iperf3 on the firewall adds additional load to the firewall that isn't fair on the test?

It starts off at a high rate, but then drops off. e.g. here's a 10 second example:

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   257 MBytes  2.16 Gbits/sec                 
[  5]   1.00-2.00   sec   266 MBytes  2.23 Gbits/sec                 
[  5]   2.00-3.00   sec   269 MBytes  2.27 Gbits/sec                 
[  5]   3.00-4.00   sec   268 MBytes  2.25 Gbits/sec                 
[  5]   4.00-5.00   sec   202 MBytes  1.69 Gbits/sec                 
[  5]   5.00-6.00   sec   116 MBytes   971 Mbits/sec                 
[  5]   6.00-7.00   sec  82.4 MBytes   689 Mbits/sec                 
[  5]   7.00-8.00   sec  91.9 MBytes   771 Mbits/sec                 
[  5]   8.00-9.00   sec   105 MBytes   880 Mbits/sec                 
[  5]   9.00-10.00  sec   101 MBytes   848 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.72 GBytes  1.48 Gbits/sec  383             sender
[  5]   0.00-10.00  sec  1.72 GBytes  1.47 Gbits/sec                  receiver

If I run the iperf3 test for 60 seconds the throughput tends to vary a lot

Also this is with the emulated netmap driver. if I configure Zenarmour to use the native netmap driver, the throughput sometimes drops to 0 for a few seconds and then recovers - it looks like maybe something crashed.

[sy]

Hi,

Please do same test in bypass mode (Dashboard - Engine - Enter Bypass Mode) as well.

[meyergru]

As for the drop in speed after a few seconds: When zenarmour is enabled and eats CPU, the PL1 kicks in after a few seconds. Yout can probably change PL1 and PL2 and possibly also the hold time in the BIOS.

But it should be noted: many of those china boxes come with abysmal fitting of heatsinks because of protruding grates or badly applied thermal paste, look here.

If your specimen has a similar problem, the performance drop may be much higher than the expected 30% because of high temps quickly developing. Your transfer rates drop to one third! Also, if CPU was not the limiting factor in the first place (which it appears), the effective drop is even lower.

I suspect that you have cooling problems which become visible through heavy throttling.

[dss]

Thanks for the suggestions.

I should have mentioned, I don't think it's a thermal issue as the 4 core temperatures don't get above 38 degrees according to the OPNSense dashboard. The device is this one which has a chunky heatsink, and then a large fan is mounted as well: https://cwwk.net/products/6-lan-firewall-appliance-2-5g-router-12th-gen-intel-i3-n305-n100-ddr5-2-nvme-2-sata3-0-fanless-mini-pc-esxi-proxmox-host

Also worth mentioning i'm running OPNsense on bare metal.

I'm not familiar with the PL1/PL2 settings. Will it switch to PL1 even if the temperature doesn't get too high?

These are the numbers with Zenarmour bypassed...

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   282 MBytes  2.36 Gbits/sec    0    966 KBytes       
[  5]   1.00-2.00   sec   279 MBytes  2.34 Gbits/sec    0   1.09 MBytes       
[  5]   2.00-3.00   sec   279 MBytes  2.34 Gbits/sec    0   1.09 MBytes       
[  5]   3.00-4.00   sec   279 MBytes  2.34 Gbits/sec    0   1.09 MBytes       
[  5]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec    0   1.09 MBytes       
[  5]   5.00-6.00   sec   275 MBytes  2.31 Gbits/sec    0   1.09 MBytes       
[  5]   6.00-7.00   sec   161 MBytes  1.35 Gbits/sec    0   1.68 MBytes       
[  5]   7.00-8.00   sec   101 MBytes   849 Mbits/sec    0   1.68 MBytes       
[  5]   8.00-9.00   sec  83.8 MBytes   703 Mbits/sec    0   1.68 MBytes       
[  5]   9.00-10.00  sec  83.8 MBytes   703 Mbits/sec    0   1.68 MBytes   

So still not hitting 2.5gbe for the duration.

[meyergru]

I'm not familiar with the PL1/PL2 settings. Will it switch to PL1 even if the temperature doesn't get too high?

Yes.

These are the numbers with Zenarmour bypassed...

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   282 MBytes  2.36 Gbits/sec    0    966 KBytes       
[  5]   1.00-2.00   sec   279 MBytes  2.34 Gbits/sec    0   1.09 MBytes       
[  5]   2.00-3.00   sec   279 MBytes  2.34 Gbits/sec    0   1.09 MBytes       
[  5]   3.00-4.00   sec   279 MBytes  2.34 Gbits/sec    0   1.09 MBytes       
[  5]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec    0   1.09 MBytes       
[  5]   5.00-6.00   sec   275 MBytes  2.31 Gbits/sec    0   1.09 MBytes       
[  5]   6.00-7.00   sec   161 MBytes  1.35 Gbits/sec    0   1.68 MBytes       
[  5]   7.00-8.00   sec   101 MBytes   849 Mbits/sec    0   1.68 MBytes       
[  5]   8.00-9.00   sec  83.8 MBytes   703 Mbits/sec    0   1.68 MBytes       
[  5]   9.00-10.00  sec  83.8 MBytes   703 Mbits/sec    0   1.68 MBytes   

So still not hitting 2.5gbe for the duration.

Are there any other components involved that explain why there is this performance hit after a few seconds?

If it is not the temperature throttling, this looks like a very low PL1, which would explain the 38°C as well. You could raise it in that case as there seems to be enough headroom.