OpenVPN performance differential (openWRT, pfSense & OPNsense)

[Ilnahro]

Hi folks,

Intro (skippable)
I recently decided to try using a self-hosted router to allow all my network traffic to be routed through my VPN provider transparently. It has been a very interesting journey so far. I started with OpenWRT, but it's stable releases were very old and the snapshots contained too many bugs to use on a daily basis. I then tried pfSense but I kept searching for alternatives and eventually stumbled upon OPNsense and it's vision and style align much better with my preference for software projects.
TL:DR: Recently started using OPNsense

I created essentially identical setups with openWRT, pfSense and OPNsense to tunnel my network traffic through my VPN. I would prefer to continue using OPNsense, however, the performance difference in terms of OpenVPN throughput is staggering:

pfSense (2.4.3) 60Mb/s
openWRT (1.17.04) 85Mb/s
OPNsense (18.1.5) 30Mb/s

For reference on my setup:
All softwares are running in a VirtualBox VM on a Windows 10 Pro host with the following specs:
CPU: Athlon X4 620 @ 3GHz
RAM: 4GB DDR3-1333
Of that, I dedicated 3 cores and 1024MB to the respective VMs and testing was done successively. Network adapters are emulated as Intel PRO/1000 MT Desktop (with the exception of OpenWRT which benefits from paravirtualized network adapters. They are not used on OPNsense and pfSense because in those two, they incur a steep performance penalty). Underlying hardware are Gbit-Realtek NICs (easily capable of pushing more than 100Mb/s consistently).
Connection using direct connection via the provider router:
Down: 100Mb/s (advertised), 90-110Mb/s (actual)
Up: 5Mb/s (advertised), 30Mb/s (actual)
The VPN provider (mullvad.net) uses AES-256-CBC to encrypt the traffic with LZO compression enabled (non-adaptive). They also provide a very complete guide to setup on openWRT and pfSense (which works for OPNsense with essentially no changes).

Now, I am not surprised that my CPU fails to achieve the maximum throughput given the usual performance of OpenVPN/OpenSSL, however, I am very surprised by the performance difference between OPNsense and pfSense. I expected a performance penalty coming from openWRT (given it's designed for embedded systems) but I expected OPNsense to perform similarly to pfSense (if not better).

So to you guys myquestion: is there something obvious in the OPNsense/OpenVPN settings that I might be missing that would massively influence the performance? Or is there a reason I should expect OPNsense to perform much worse in combination with OpenVPN?

Any tips or ideas would be greatly appreciated  :D

[mimugmail]

I would disable compression, only makes sense on high latency or low band links

[elektroinside]

I'm often maxing out my upload (~500MB) over OpenVPN if i connect from another 1GB link.

This might help: https://forum.opnsense.org/index.php?topic=6590.0

[Ilnahro]

I'm often maxing out my upload (~500MB) over OpenVPN if i connect from another 1GB link.

This might help: https://forum.opnsense.org/index.php?topic=6590.0

That's a very interesting thread. I'll have a look at what of that I could apply in my situation tomorrow.
I am not sure I am suffering from the same problem (I do not seem to incur a loss in the quality of the connection, only in the bandwidth) and my speeds are consistent (if slower than expected). But I'm new to this, so I might be way off  8)

I will also give disabling compression a go then as well.

Thanks for the quick and helpful responses :)

PS: Never would have occurred to me to look in the intrusion section for performance improvements  ::)

Edit: Seems that disabling compression is not an option with my VPN provider. They appear to enforce compression. The connection does get established but it does not allow any network access and the log gets filled with the following message:

Code Select Expand

openvpn[22303]: Bad compression stub decompression header byte: 102
I half expected this outcome given that enabling always-on compression is an explicit part of their tutorials and is found in all their OpenVPN configs, but it was still worth a try. They are very responsive in terms of support, so I will send them a message about using no compression or at least adaptive compression. Who knows.

[opnfwb]

In OPNsense, go to Interfaces/Settings. I believe by default, OPNsense has Hardware CRC, Hardware TSO, and Hardware LRO all disabled.

I have not used pfSense in a few years but, I recall they used to leave some of these enabled. Perhaps that could be influencing the results a bit? However since you stated all tests are done within VMs with similar hardware allocated to the VMs, this doesn't fully make sense to me. But, it may be worth checking. Other than that I'm not sure what else it could be.

[elektroinside]

PS: Never would have occurred to me to look in the intrusion section for performance improvements  ::)

dcol wrote those with IDPS performance enhancement in mind, but from my tests, had a significant impact on OpenVPN as well. I since deleted any custom OpenVPN settings, because:
1. They didn't help much (if at all)
2. I don't need them, since dcol's settings, OpenVPN works brilliantly, with or without IDPS enabled (better if IDPS is disabled, of course, which is absolutely normal)

[Ilnahro]

In OPNsense, go to Interfaces/Settings. I believe by default, OPNsense has Hardware CRC, Hardware TSO, and Hardware LRO all disabled.

I have not used pfSense in a few years but, I recall they used to leave some of these enabled. Perhaps that could be influencing the results a bit? However since you stated all tests are done within VMs with similar hardware allocated to the VMs, this doesn't fully make sense to me. But, it may be worth checking. Other than that I'm not sure what else it could be.

True, those are all disabled. In both pfSense and OPNsense. I'll check if it has an impact, but in openWRT toggling them did not. It might offload some of the CPU load from the VM to the host.

Edit: They do not appear to make a measurable difference.

[franco]

It's not noted here... have you by any chance tested OPNsense on LibreSSL? Because we just recently discovered it started to silently compile without AES-NI support:

https://github.com/opnsense/core/issues/2343


Cheers,
Franco

[john9527]

Is there possibly something similar going on with OpenSSL.    I have a 100Mbps connection which benchmarks at about 120Mbps without OpenVPN active.

Turning on OpenVPN I get the following results with the same settings
- System HW crypto set to AES-NI
- OpenVPN HW crypto set to Intel RDRAND

pfSense (2.4.3): 100-110Mbps
OPNSense (18.1.6): 75-80Mbps

I see in the logs that my processor (N3700) is recognized as AES-NI capable.   Turning off the crypto options makes no difference on OPNSense, so it appears that the aesni acceleration isn't being used.

[franco]

> - OpenVPN HW crypto set to Intel RDRAND

Are you sure this is correct? As far as I know it needs no setting at all for AES-NI.


Cheers,
Franco

[john9527]

Two options...None and RDRAND.   Tried both ways with no difference.   pfSense measures taken with it set to RDRAND.

[franco]

That makes little sense... it would indicate the cipher you are using is not covered by AES-NI, but only RDRAND.

I checked OpenSSL, AES-NI works. But I don't have an OpenVPN to test speed at the moment.


Cheers,
Franco

[john9527]

Using AES-128-CBC, SHA1 for OpenVPN

Just to make sure we understand, the RDRAND option is making no difference in the results.   Same with turning off AES-NI in system settings.   Something is definitely broke.

BTW, pfSense shows the CPU crypto options in the dashboard, along with their state (active or inactive).   Would be a good addition to OPNSense.

[franco]

Thanks for the clarification. Could be, but unsure where to look for further clues. AES-NI is quite elusive and questions tend to come in in waves.


Cheers,
Franco

[john9527]

Agreed, always a bit of black magic involved.

I'm trying to sort through things as well.   It would seem that the aesni module doesn't need to be loaded unless the old-crypto option is selected, and that's what I'm seeing.  I haven't had the opportunity yet to trace what the RDRAND option is setting in the code.