OPNsense Forum

English Forums => Zenarmor (Sensei) => Topic started by: fakebizprez on May 01, 2025, 10:57:20 AM

Title: Throughput Getting Crushed
Post by: fakebizprez on May 01, 2025, 10:57:20 AM
My download has gone from 5.2Gbps to 3.2Gbps & Upload has gone from 5.1Gbps to 1.4Gbps since installing Zenarmor, and I have OPNsense running on a Dell PowerEdge R730 with on two Intel Xeon E5-2643 v3 @ 3.40GHz (6 cores each). 64GB RAM.


Is this to be expected or have I misconfigured the extension?
Title: Re: Throughput Getting Crushed
Post by: meyergru on May 01, 2025, 11:22:09 AM
I would have expected that (https://forum.opnsense.org/index.php?topic=41295.0) with use of a single thread only for the free tier, but I do not use Zenarmor.
Title: Re: Throughput Getting Crushed
Post by: Seimus on May 01, 2025, 01:48:18 PM
Yop, this is the expected throughput in regards of your CPU when using ZA.
I would even say you are bit above the expected.

Regards,
S.
Title: Re: Throughput Getting Crushed
Post by: fakebizprez on May 02, 2025, 02:14:34 PM
Thanks for the replies.

Only the free tier is single-threaded, am I understanding that correctly?
Title: Re: Throughput Getting Crushed
Post by: Seimus on May 02, 2025, 03:43:14 PM
There is no multicore support yet.

As for when there will be and on which subscription is still not properly communicated by ZA. There is a Major thread about this topic.

Regards,
S.
Title: Re: Throughput Getting Crushed
Post by: fakebizprez on May 04, 2025, 04:25:07 AM
Thank you for answering

Quote from: Seimus on May 02, 2025, 03:43:14 PMThere is a Major thread about this topic.

I read through it. Looks like this has been in progress for an very long time. Unfortunately, I cannot wait while continuing to use the product.

Aside from it being single-threaded, are there any adjustments that can be made to reduce the reduction in throughput? For example, should I use something other than ElasticSearch? Should I not emulate the Netmap driver? Should I not include Wireguard? Anything at all..
Title: Re: Throughput Getting Crushed
Post by: Seimus on May 04, 2025, 06:31:47 PM
The only way how to improve performance for ZA is to move of the DB to an external one.

Regards,
S.
Title: Re: Throughput Getting Crushed
Post by: sy on May 05, 2025, 12:40:57 PM
Hi,

Good news regarding multicore support. We have released a test binary. Kindly contact the support team via the "Have Feedback" option located in the bottom right corner of the UI if you wish to try it out.

Title: Re: Throughput Getting Crushed
Post by: fakebizprez on May 09, 2025, 02:36:22 AM
Quote from: sy on May 05, 2025, 12:40:57 PMHi,

Good news regarding multicore support. We have released a test binary. Kindly contact the support team via the "Have Feedback" option located in the bottom right corner of the UI if you wish to try it out.



Thank you, I will do this now.
Title: Re: Throughput Getting Crushed
Post by: Taunt9930 on May 12, 2025, 08:03:51 PM
Quote from: fakebizprez on May 09, 2025, 02:36:22 AM
Quote from: sy on May 05, 2025, 12:40:57 PMHi,

Good news regarding multicore support. We have released a test binary. Kindly contact the support team via the "Have Feedback" option located in the bottom right corner of the UI if you wish to try it out.



Thank you, I will do this now.

Has it made a difference in your use-case?
Title: Re: Throughput Getting Crushed
Post by: fakebizprez on May 13, 2025, 06:42:04 AM
Quote from: Taunt9930 on May 12, 2025, 08:03:51 PM
Quote from: fakebizprez on May 09, 2025, 02:36:22 AM
Quote from: sy on May 05, 2025, 12:40:57 PMHi,

Good news regarding multicore support. We have released a test binary. Kindly contact the support team via the "Have Feedback" option located in the bottom right corner of the UI if you wish to try it out.



Thank you, I will do this now.

Has it made a difference in your use-case?

I just set it up, per the instructions that were emailed to me, but there seem to be errors. I forwarded the logs to the ZenArmor team.

When they get back to me tomorrow I will troubleshoot and report back.
Title: Re: Throughput Getting Crushed
Post by: Lurick on May 15, 2025, 01:03:30 PM
Quote from: fakebizprez on May 13, 2025, 06:42:04 AM
Quote from: Taunt9930 on May 12, 2025, 08:03:51 PM
Quote from: fakebizprez on May 09, 2025, 02:36:22 AM
Quote from: sy on May 05, 2025, 12:40:57 PMHi,

Good news regarding multicore support. We have released a test binary. Kindly contact the support team via the "Have Feedback" option located in the bottom right corner of the UI if you wish to try it out.



Thank you, I will do this now.

Has it made a difference in your use-case?

I just set it up, per the instructions that were emailed to me, but there seem to be errors. I forwarded the logs to the ZenArmor team.

When they get back to me tomorrow I will troubleshoot and report back.

Similar boat for me. Tons of issues with IPv4 connectivity as well. Hopefully they can get some of this resolved :)
Title: Re: Throughput Getting Crushed
Post by: Seimus on May 17, 2025, 10:14:55 AM
I actually remembered.

For the time being, You can improve ZA performance by using RSS + Do not pin Engine packet processor to dedicated CPU.

If you set it right you should be in theory able to go bit above. Helped me on a N100 CPU to go from 1Gbit/s to around ~1.8Gbit.

Regards,
S.
Title: Re: Throughput Getting Crushed
Post by: fakebizprez on May 29, 2025, 09:50:26 PM
I was not using the Pin feature.

Off the top of my head I am not sure what RSS is.
Title: Re: Throughput Getting Crushed
Post by: fakebizprez on May 29, 2025, 09:56:41 PM
Quote from: Lurick on May 15, 2025, 01:03:30 PM
Quote from: fakebizprez on May 13, 2025, 06:42:04 AM
Quote from: Taunt9930 on May 12, 2025, 08:03:51 PM
Quote from: fakebizprez on May 09, 2025, 02:36:22 AM
Quote from: sy on May 05, 2025, 12:40:57 PMHi,

Good news regarding multicore support. We have released a test binary. Kindly contact the support team via the "Have Feedback" option located in the bottom right corner of the UI if you wish to try it out.



Thank you, I will do this now.

Has it made a difference in your use-case?

I just set it up, per the instructions that were emailed to me, but there seem to be errors. I forwarded the logs to the ZenArmor team.

When they get back to me tomorrow I will troubleshoot and report back.

Similar boat for me. Tons of issues with IPv4 connectivity as well. Hopefully they can get some of this resolved :)

I don't have the logs available right now, because I'm on my phone, but the multi-threaded throughput was actually lower than single-threaded.

Unfortunately, I had to uninstall the service until they can polish this new feature. My OPNsense server's two CPUs have the highest rates single-threaded performance available for the PowerEdge R730.
Title: Re: Throughput Getting Crushed
Post by: Seimus on May 30, 2025, 06:57:46 PM
Quote from: fakebizprez on May 29, 2025, 09:50:26 PMOff the top of my head I am not sure what RSS is.

https://docs.opnsense.org/troubleshooting/performance.html#receive-side-scaling
Title: Re: Throughput Getting Crushed
Post by: fakebizprez on May 31, 2025, 02:05:56 PM
Quote from: Seimus on May 30, 2025, 06:57:46 PMhttps://docs.opnsense.org/troubleshooting/performance.html#receive-side-scaling
Wow. Interesting..........

So there's three options: RSS, Unpin CPU, & push logs to remote database, correct?
Title: Re: Throughput Getting Crushed
Post by: Seimus on June 01, 2025, 11:17:51 AM
Correct,

I use RSS + CPU unpin for like 2y without problem and having ~ 1.7G is much better than 1G as I need high throughput for interVLAN communication rather than LAN to WAN.

Moving elastic or other DB type depending what you use to a remote one can lift the performance too cause it will not eat into the FWs resources.

And one last tip, depending on your deployment of ZA, always deploy it on the Parent interface not per interface. Each single interface spans additional eastpect process. Meaning I use a LAGG on which I have dozen of VLANs, I do not run ZA on those VLANs I run it on the LAGG. Thus I have only single eastpect process.

Iperf test --- InterVLAN only = 1668Mbit/s
4 different host, cross InterVLAN combined at the same time. This is post RSS + unpin CPU in ZA

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.04  sec  5.57 GBytes   797 Mbits/sec  691            sender
[  5]   0.00-60.00  sec  5.56 GBytes   797 Mbits/sec                  receiver

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  6.09 GBytes   871 Mbits/sec  989             sender
[  5]   0.00-60.00  sec  6.08 GBytes   871 Mbits/sec                  receiver

Regards,
S.

Title: Re: Throughput Getting Crushed
Post by: fakebizprez on June 02, 2025, 04:05:24 AM
OK, that's great information, man, I'm going to have to copy/paste this in my workspace. My company is scattered throughout the globe, and there's zero humans on my LAN, aside from me, so I was considering just putting it on WG0 (wireguard) because I'm hosting a fairly substantial VPN server. They're the ones that need the protection of a NGFW the most.
Title: Re: Throughput Getting Crushed
Post by: Seimus on June 02, 2025, 10:07:50 AM
Quote from: fakebizprez on June 02, 2025, 04:05:24 AMso I was considering just putting it on WG0 (wireguard)
You can do that, ZA works on WG tunnels as well if you created it as an Interface and assigned it.

Regards,
S.