Just go a HUNSN RJ42 in (shipped from Amazon Germany, https://www.amazon.de/dp/B0C985FVT1 ).
Installed Proxmox and passed through two NICs to an OPNsense VM.
Without Zenarmor, full 2.5G throughput, measured through the box with a local 10G iperf3 server on my WAN.
With Zenarmor Free edition (NICs are in L3 with native netmap driver, seems to work fine) it looks like this
iperf3 -R -t60
[ 5] 0.00-60.04 sec 14.7 GBytes 2.10 Gbits/sec 1957 sender
[ 5] 0.00-60.00 sec 14.7 GBytes 2.10 Gbits/sec receiver
iper3 -t60
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 10.3 GBytes 1.48 Gbits/sec 3800 sender
[ 5] 0.00-60.04 sec 10.3 GBytes 1.48 Gbits/sec receiver
Awesome! :)
Hi @athurdent,
What about with emulated netmap driver?
Hi @sy,
looks like this with the emulated driver
iperf3 -R -t60
[ 5] 0.00-60.04 sec 8.93 GBytes 1.28 Gbits/sec 1245 sender
[ 5] 0.00-60.00 sec 8.93 GBytes 1.28 Gbits/sec receiver
iper3 -t60
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 10.8 GBytes 1.55 Gbits/sec 2117 sender
[ 5] 0.00-60.04 sec 10.8 GBytes 1.54 Gbits/sec receiver
And here's the blind test, no Zenarmor, forgot it in the OP.
iperf3 -R -t60
[ 5] 0.00-60.04 sec 16.4 GBytes 2.35 Gbits/sec 842 sender
[ 5] 0.00-60.00 sec 16.4 GBytes 2.35 Gbits/sec receiver
iper3 -t60
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 16.4 GBytes 2.35 Gbits/sec 1776 sender
[ 5] 0.00-60.04 sec 16.4 GBytes 2.35 Gbits/sec receiver
Here's a fresh test with a MacBook Pro using a 2.5G adapter, this time with 5 concurrent streams.
I'm getting line rate throughput, fantastic!
% iperf3-darwin -c192.168.178.8 -R -P5 -t60
...
[SUM] 0.00-60.00 sec 16.1 GBytes 2.30 Gbits/sec 22304 sender
[SUM] 0.00-60.00 sec 16.0 GBytes 2.30 Gbits/sec receiver
% iperf3-darwin -c192.168.178.8 -P5 -t60
...
[SUM] 0.00-60.00 sec 16.2 GBytes 2.32 Gbits/sec 5542680 sender
[SUM] 0.00-60.01 sec 16.2 GBytes 2.32 Gbits/sec receiver
Any chance you could test with the same config, but virtial NICs (Virtio - default options) in the OPNsense VM vs passthrough? One with and one without Zenarmor would be awesome. I'm getting a similar machine soon and have always run OPNsense with virtualized NICs. Thanks.
An important update (multicore eastpect) is expected in October. So, wait with your measurements until that update is published.
This reduction in throughput can be attributed to various factors related to the configuration and performance of Zenarmor (OPNsenseGeometry Dash (https://geometrydashlite.io)) within your Proxmox virtualized environment.
Quote from: Keaton Mertz on December 22, 2023, 04:22:19 AM
This reduction in throughput can be attributed to various factors related to the configuration and performance of Zenarmor (OPNsenseGeometry Dash (https://geometrydashlite.io)) within your Proxmox virtualized environment.
I would not call line rate throughput a reduction... 😉
Quote from: Keaton Mertz on December 22, 2023, 04:22:19 AM
This reduction in throughput can be attributed to various factors related to the configuration and performance of Zenarmor (OPNsenseGeometry Dash (https://geometrydashlite.io)) within your Proxmox virtualized environment.
run 3 (https://run-3.pro/)
I also agree with this point of view. Many answers to the problem are being raised. I'm doing a lot of research and testing.
What's the best way to set up iperf3 for testing OPNsense/Zenarmour throughput? I've got a CWWK N100 / i226v firewall and if I run iperf3 as a server on the firewall and plug a macbook pro with a 2.5gbe into the LAN port, I'm not getting close to 2.5Gbe speeds when Zenarmour is enabled. But perhaps running iperf3 on the firewall adds additional load to the firewall that isn't fair on the test?
It starts off at a high rate, but then drops off. e.g. here's a 10 second example:
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 257 MBytes 2.16 Gbits/sec
[ 5] 1.00-2.00 sec 266 MBytes 2.23 Gbits/sec
[ 5] 2.00-3.00 sec 269 MBytes 2.27 Gbits/sec
[ 5] 3.00-4.00 sec 268 MBytes 2.25 Gbits/sec
[ 5] 4.00-5.00 sec 202 MBytes 1.69 Gbits/sec
[ 5] 5.00-6.00 sec 116 MBytes 971 Mbits/sec
[ 5] 6.00-7.00 sec 82.4 MBytes 689 Mbits/sec
[ 5] 7.00-8.00 sec 91.9 MBytes 771 Mbits/sec
[ 5] 8.00-9.00 sec 105 MBytes 880 Mbits/sec
[ 5] 9.00-10.00 sec 101 MBytes 848 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 1.72 GBytes 1.48 Gbits/sec 383 sender
[ 5] 0.00-10.00 sec 1.72 GBytes 1.47 Gbits/sec receiver
If I run the iperf3 test for 60 seconds the throughput tends to vary a lot
Also this is with the emulated netmap driver. if I configure Zenarmour to use the native netmap driver, the throughput sometimes drops to 0 for a few seconds and then recovers - it looks like maybe something crashed.
Hi,
Please do same test in bypass mode (Dashboard - Engine - Enter Bypass Mode) as well.
As for the drop in speed after a few seconds: When zenarmour is enabled and eats CPU, the PL1 kicks in after a few seconds. Yout can probably change PL1 and PL2 and possibly also the hold time in the BIOS.
But it should be noted: many of those china boxes come with abysmal fitting of heatsinks because of protruding grates or badly applied thermal paste, look here (https://www.congenio.de/infos/opnsense-hardware.html).
If your specimen has a similar problem, the performance drop may be much higher than the expected 30% because of high temps quickly developing. Your transfer rates drop to one third! Also, if CPU was not the limiting factor in the first place (which it appears), the effective drop is even lower.
I suspect that you have cooling problems which become visible through heavy throttling.
Thanks for the suggestions.
I should have mentioned, I don't think it's a thermal issue as the 4 core temperatures don't get above 38 degrees according to the OPNSense dashboard. The device is this one which has a chunky heatsink, and then a large fan is mounted as well: https://cwwk.net/products/6-lan-firewall-appliance-2-5g-router-12th-gen-intel-i3-n305-n100-ddr5-2-nvme-2-sata3-0-fanless-mini-pc-esxi-proxmox-host
Also worth mentioning i'm running OPNsense on bare metal.
I'm not familiar with the PL1/PL2 settings. Will it switch to PL1 even if the temperature doesn't get too high?
These are the numbers with Zenarmour bypassed...
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 282 MBytes 2.36 Gbits/sec 0 966 KBytes
[ 5] 1.00-2.00 sec 279 MBytes 2.34 Gbits/sec 0 1.09 MBytes
[ 5] 2.00-3.00 sec 279 MBytes 2.34 Gbits/sec 0 1.09 MBytes
[ 5] 3.00-4.00 sec 279 MBytes 2.34 Gbits/sec 0 1.09 MBytes
[ 5] 4.00-5.00 sec 280 MBytes 2.35 Gbits/sec 0 1.09 MBytes
[ 5] 5.00-6.00 sec 275 MBytes 2.31 Gbits/sec 0 1.09 MBytes
[ 5] 6.00-7.00 sec 161 MBytes 1.35 Gbits/sec 0 1.68 MBytes
[ 5] 7.00-8.00 sec 101 MBytes 849 Mbits/sec 0 1.68 MBytes
[ 5] 8.00-9.00 sec 83.8 MBytes 703 Mbits/sec 0 1.68 MBytes
[ 5] 9.00-10.00 sec 83.8 MBytes 703 Mbits/sec 0 1.68 MBytes
So still not hitting 2.5gbe for the duration.
Quote from: dss on March 26, 2024, 08:04:27 PM
I'm not familiar with the PL1/PL2 settings. Will it switch to PL1 even if the temperature doesn't get too high?
Yes.
Quote from: dss on March 26, 2024, 08:04:27 PM
These are the numbers with Zenarmour bypassed...
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 282 MBytes 2.36 Gbits/sec 0 966 KBytes
[ 5] 1.00-2.00 sec 279 MBytes 2.34 Gbits/sec 0 1.09 MBytes
[ 5] 2.00-3.00 sec 279 MBytes 2.34 Gbits/sec 0 1.09 MBytes
[ 5] 3.00-4.00 sec 279 MBytes 2.34 Gbits/sec 0 1.09 MBytes
[ 5] 4.00-5.00 sec 280 MBytes 2.35 Gbits/sec 0 1.09 MBytes
[ 5] 5.00-6.00 sec 275 MBytes 2.31 Gbits/sec 0 1.09 MBytes
[ 5] 6.00-7.00 sec 161 MBytes 1.35 Gbits/sec 0 1.68 MBytes
[ 5] 7.00-8.00 sec 101 MBytes 849 Mbits/sec 0 1.68 MBytes
[ 5] 8.00-9.00 sec 83.8 MBytes 703 Mbits/sec 0 1.68 MBytes
[ 5] 9.00-10.00 sec 83.8 MBytes 703 Mbits/sec 0 1.68 MBytes
So still not hitting 2.5gbe for the duration.
Are there any other components involved that explain why there is this performance hit after a few seconds?
If it is not the temperature throttling, this looks like a very low PL1, which would explain the 38°C as well. You could raise it in that case as there seems to be enough headroom.
So, frustratingly there's no PL1/PL2 options in the BIOS :( I've seen Linux scripts that allow software configuration of this, but I can't find a FreeBSD equivalent.
I've been doing a little more reading about PL1/PL2 settings and from what I understand, typically the default TAU values are maybe half a minute or even closer to 1 minute. What I'm seeing on my router is that the 2.5G throughput is only maintained for 7-8 seconds. That leaves me a little doubtful it's due to PL1 kicking in?
Also I noticed that when running Zenarmour in bypass mode, there is a still an eastpect process running and it's single threaded. After Stopping Zenarmour that process disappears. I'm still not getting the full 2.5G even when Zenarmour is stopped though. What's the intended difference between Bypass and Stopped modes?
For their N5105 line, these options existed: my unit was from HUNSN (CWWK rebranded), so I flashed the BIOS from CWWK.
You can ask their support if they have a more open version. Also, if you have the BIOS file, you can modify it with tools available on the internet to enable such options. This is risky, though.
Some advice if you want to flash third party BIOS:
- buy an EEPROM programmer;
- check if it supports your chip;
- backup original BIOS (save it to the cloud).
Just to add, some really odd things going on with my N100 CWWK box...
When under load the core temperatures actually drop according the OPNsense monitoring. At idle it's around 34-36°C. Under load it initially hits 41°C for a few seconds then drops to 30°C.
I can see from the following command that CPU frequency starts at 2923mhz, then drops to 800mhz or even down to 400mhz under load.
$ sysctl -a | grep dev.cpu.*.freq:
dev.cpu.3.freq: 402
dev.cpu.2.freq: 402
dev.cpu.1.freq: 402
dev.cpu.0.freq: 402
So something is definitely applying a throttle. Perhaps it is that PL1 is configured way to low, but I can't even see what the value of that is in my set up.
I've also spotted that Zenarmour's eastpect process disappears and then reappears when under load. The Zenarmour dashboard still reports it as 'running' when this happens.
When Zenarmour's eastpect process is running and the CPU's drop down below 800mhz, my throughput is severely limited (e.g. 300-500 mbps). This is a problem for my internet connection which is 1Gbps symmetrical.
For now I'm going to turn off Zenarmour until I've worked out what the issue is with the hardware.
I've given up trying to get this 6-port CWWK N100 box to perform. No one seems to have a BIOS for it that opens up the PL1/PL2 settings. I'm going to return it to the original supplier.
I've now taken a chance and purchased a 4-port CWWK N100 box from a different supplier. I used Hunsn on Amazon as I'd seen other people get good results with this. Hunsn label it as an RJ35 model but it would appear to be just a CWWK rebadge like Topton and others do.
With this box I'm getting full throughput even with Zenarmour enabled. The CPU runs at around 2900mhz for the duration of the load test, unlike my previous box which dropped to 400-800mhz under load. Temps get up to about 45°C which is ok. Also the BIOS is much more open on this RJ35 unit. I'm not sure if all 4-port CWWK rebadges offer this level of BIOS control or if it's a BIOS Hunsn load to open it up.
This is the performance with Zenarmour enabled using the native netmap driver and hooked up on a 2.5GbE networking...
iperf3 -c [IP] -t 60
[ 5] 0.00-60.00 sec 16.1 GBytes 2.30 Gbits/sec 842 sender
[ 5] 0.00-60.00 sec 16.1 GBytes 2.30 Gbits/sec receiver
iperf3 -c [IP] -t 60 -R
[ 5] 0.00-60.00 sec 15.9 GBytes 2.27 Gbits/sec 4968 sender
[ 5] 0.00-60.00 sec 15.9 GBytes 2.27 Gbits/sec receiver
So I think this is similar to what @athurdent was getting.
With the emulated netmap driver the performance does drop off quite a bit to around 1.5 Gbits/sec. Also there are strange messages in the dmesg logs about the netmap driver dropping in and out when using the emulated netmap driver.