OPNsense Forum
English Forums => Hardware and Performance => Topic started by: Ilnahro on April 09, 2018, 08:45:48 pm
-
Hi folks,
Intro (skippable)
I recently decided to try using a self-hosted router to allow all my network traffic to be routed through my VPN provider transparently. It has been a very interesting journey so far. I started with OpenWRT, but it's stable releases were very old and the snapshots contained too many bugs to use on a daily basis. I then tried pfSense but I kept searching for alternatives and eventually stumbled upon OPNsense and it's vision and style align much better with my preference for software projects.
TL:DR: Recently started using OPNsense
I created essentially identical setups with openWRT, pfSense and OPNsense to tunnel my network traffic through my VPN. I would prefer to continue using OPNsense, however, the performance difference in terms of OpenVPN throughput is staggering:
pfSense (2.4.3) 60Mb/s
openWRT (1.17.04) 85Mb/s
OPNsense (18.1.5) 30Mb/s
For reference on my setup:
All softwares are running in a VirtualBox VM on a Windows 10 Pro host with the following specs:
CPU: Athlon X4 620 @ 3GHz
RAM: 4GB DDR3-1333
Of that, I dedicated 3 cores and 1024MB to the respective VMs and testing was done successively. Network adapters are emulated as Intel PRO/1000 MT Desktop (with the exception of OpenWRT which benefits from paravirtualized network adapters. They are not used on OPNsense and pfSense because in those two, they incur a steep performance penalty). Underlying hardware are Gbit-Realtek NICs (easily capable of pushing more than 100Mb/s consistently).
Connection using direct connection via the provider router:
Down: 100Mb/s (advertised), 90-110Mb/s (actual)
Up: 5Mb/s (advertised), 30Mb/s (actual)
The VPN provider (mullvad.net) uses AES-256-CBC to encrypt the traffic with LZO compression enabled (non-adaptive). They also provide a very complete guide to setup on openWRT and pfSense (which works for OPNsense with essentially no changes).
Now, I am not surprised that my CPU fails to achieve the maximum throughput given the usual performance of OpenVPN/OpenSSL, however, I am very surprised by the performance difference between OPNsense and pfSense. I expected a performance penalty coming from openWRT (given it's designed for embedded systems) but I expected OPNsense to perform similarly to pfSense (if not better).
So to you guys myquestion: is there something obvious in the OPNsense/OpenVPN settings that I might be missing that would massively influence the performance? Or is there a reason I should expect OPNsense to perform much worse in combination with OpenVPN?
Any tips or ideas would be greatly appreciated :D
-
I would disable compression, only makes sense on high latency or low band links
-
I'm often maxing out my upload (~500MB) over OpenVPN if i connect from another 1GB link.
This might help: https://forum.opnsense.org/index.php?topic=6590.0
-
I'm often maxing out my upload (~500MB) over OpenVPN if i connect from another 1GB link.
This might help: https://forum.opnsense.org/index.php?topic=6590.0
That's a very interesting thread. I'll have a look at what of that I could apply in my situation tomorrow.
I am not sure I am suffering from the same problem (I do not seem to incur a loss in the quality of the connection, only in the bandwidth) and my speeds are consistent (if slower than expected). But I'm new to this, so I might be way off 8)
I will also give disabling compression a go then as well.
Thanks for the quick and helpful responses :)
PS: Never would have occurred to me to look in the intrusion section for performance improvements ::)
Edit: Seems that disabling compression is not an option with my VPN provider. They appear to enforce compression. The connection does get established but it does not allow any network access and the log gets filled with the following message:
openvpn[22303]: Bad compression stub decompression header byte: 102I half expected this outcome given that enabling always-on compression is an explicit part of their tutorials and is found in all their OpenVPN configs, but it was still worth a try. They are very responsive in terms of support, so I will send them a message about using no compression or at least adaptive compression. Who knows.
-
In OPNsense, go to Interfaces/Settings. I believe by default, OPNsense has Hardware CRC, Hardware TSO, and Hardware LRO all disabled.
I have not used pfSense in a few years but, I recall they used to leave some of these enabled. Perhaps that could be influencing the results a bit? However since you stated all tests are done within VMs with similar hardware allocated to the VMs, this doesn't fully make sense to me. But, it may be worth checking. Other than that I'm not sure what else it could be.
-
PS: Never would have occurred to me to look in the intrusion section for performance improvements ::)
dcol wrote those with IDPS performance enhancement in mind, but from my tests, had a significant impact on OpenVPN as well. I since deleted any custom OpenVPN settings, because:
1. They didn't help much (if at all)
2. I don't need them, since dcol's settings, OpenVPN works brilliantly, with or without IDPS enabled (better if IDPS is disabled, of course, which is absolutely normal)
-
In OPNsense, go to Interfaces/Settings. I believe by default, OPNsense has Hardware CRC, Hardware TSO, and Hardware LRO all disabled.
I have not used pfSense in a few years but, I recall they used to leave some of these enabled. Perhaps that could be influencing the results a bit? However since you stated all tests are done within VMs with similar hardware allocated to the VMs, this doesn't fully make sense to me. But, it may be worth checking. Other than that I'm not sure what else it could be.
True, those are all disabled. In both pfSense and OPNsense. I'll check if it has an impact, but in openWRT toggling them did not. It might offload some of the CPU load from the VM to the host.
Edit: They do not appear to make a measurable difference.
-
It's not noted here... have you by any chance tested OPNsense on LibreSSL? Because we just recently discovered it started to silently compile without AES-NI support:
https://github.com/opnsense/core/issues/2343
Cheers,
Franco
-
Is there possibly something similar going on with OpenSSL. I have a 100Mbps connection which benchmarks at about 120Mbps without OpenVPN active.
Turning on OpenVPN I get the following results with the same settings
- System HW crypto set to AES-NI
- OpenVPN HW crypto set to Intel RDRAND
pfSense (2.4.3): 100-110Mbps
OPNSense (18.1.6): 75-80Mbps
I see in the logs that my processor (N3700) is recognized as AES-NI capable. Turning off the crypto options makes no difference on OPNSense, so it appears that the aesni acceleration isn't being used.
-
> - OpenVPN HW crypto set to Intel RDRAND
Are you sure this is correct? As far as I know it needs no setting at all for AES-NI.
Cheers,
Franco
-
Two options...None and RDRAND. Tried both ways with no difference. pfSense measures taken with it set to RDRAND.
-
That makes little sense... it would indicate the cipher you are using is not covered by AES-NI, but only RDRAND.
I checked OpenSSL, AES-NI works. But I don't have an OpenVPN to test speed at the moment.
Cheers,
Franco
-
Using AES-128-CBC, SHA1 for OpenVPN
Just to make sure we understand, the RDRAND option is making no difference in the results. Same with turning off AES-NI in system settings. Something is definitely broke.
BTW, pfSense shows the CPU crypto options in the dashboard, along with their state (active or inactive). Would be a good addition to OPNSense.
-
Thanks for the clarification. Could be, but unsure where to look for further clues. AES-NI is quite elusive and questions tend to come in in waves.
Cheers,
Franco
-
Agreed, always a bit of black magic involved.
I'm trying to sort through things as well. It would seem that the aesni module doesn't need to be loaded unless the old-crypto option is selected, and that's what I'm seeing. I haven't had the opportunity yet to trace what the RDRAND option is setting in the code.
-
Thanks for the clarification. Could be, but unsure where to look for further clues. AES-NI is quite elusive and questions tend to come in in waves.
Cheers,
Franco
Well, you are not going to believe this one. After checking everything I could think of (the code was handling all the options correctly, the makefiles looked correct, etc), I thought could something be limiting the processor. I was running with PowerD disabled.....I enabled HiAdaptive mode and miraculously the speed came up right in line with my pfSense measurements! Not sure I understand, but I'll take it :)
-
Yay, great work figuring that out. Happy to hear. The tinkerers will be interested as well always trying to squeeze out more performance... :)
Cheers,
Franco
-
I regret falling silent in the middle of this discussion (I had to move abroad), but I finally got around to tinkering with this problem again and I am happy to report I actually seem to have found a solution.
My setup has changed somewhat (now running an FX8300 (with AES-NI) and 16GB as the host for the VM) as has my connection (now 200/50), but I was experiencing the same problem (without OpenVPN, OPNSense managed the full 200/50, with OpenVPN I was stuck at very inconsistent speeds which averaged around 20/20 and maxed out at 30/30 occasionally even after extensive tweaking).
The limiting factor appears to have been the send/receive buffers. I am not sure what default values OpenVPN applies in OPNSense, but I found a thread (https://forum.netgate.com/topic/103216/pfsense-hardware-for-home-router-openvpn-performance (https://forum.netgate.com/topic/103216/pfsense-hardware-for-home-router-openvpn-performance)) about pfsense discussing OpenVPN performance where someone posted their config including the buffers. Adding them immediately boosted my Download to the expected 200 on download. My download is still stuck around 20-30 but it seems like this is more a limitation of my VPN provider (as I cant get over 30Mbits for upload on my Windows Desktop either)
TL:DR
Add the following lines to the advanced config for OpenVPN connections:
sndbuf 524288;
rcvbuf 524288
Maybe this will help someone who read this thread and couldn't get any further than I had come so far :)
-
Yeah the first issue is obvious, your using an emulated e1000 driver.
Even if there is a penalty using a paravirtualized driver it should still outperform the emulated driver.
So for something that's actually comparative, you need to provide the results of OpenWrt running with the e1000.
Of course for the vmxnet driver to work you need to also ensure the same version of vmtools is installed to all guests to ensure the vmxnet driver being compared is identical.
Hope this helps.
-
Whilst I think about it the other issue is your running it on a windows 10 OS with virtualbox, and with all good intention you cant ring fence the network traffic of the win10 OS, such broadcasting, antivirus traffic, etc
You should try running it on a monolithic hypervisor for a truer comparison.