OpenVPN seems not to use AES-NI

[roschi]

Hi all,

I know that there are some threads discussing OpenVPN performance already, but most problems ended up to be caused by bad configurations and I don't think that's the problem in my case.

My setup is as follows:


I've got an OpenVPN server running on a quite powerful VPS (host-passthrough of two Xeon E5-2680v4 cores) which performs really well. The OS is Debian 9. OpenSSL benchmarking shows following results on that VPS:

VPS $ openssl speed -elapsed -evp aes-256-gcm
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-gcm for 3s on 16 size blocks: 48248172 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 64 size blocks: 35021689 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 256 size blocks: 18801677 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 1024 size blocks: 6310897 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 8192 size blocks: 927724 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 16384 size blocks: 473874 aes-256-gcm's in 3.00s
OpenSSL 1.1.0f  25 May 2017
built on: reproducible build, date unspecified
options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr)
compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/lib/ssl\"" -DENGINESDIR="\"/usr/lib/x86_64-linux-gnu/engines-1.1\""
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm     257323.58k   747129.37k  1604409.77k  2154119.51k  2533305.00k  2587983.87k

Then, there is an OpenVPN client on an OPNsense running locally as KVM guest on a Pentium J3710 with all 4 cores passed through. OpenSSL benchmarks on that OPNsense are as follows:

root@OPNsense:~ # /usr/local/bin/openssl speed -elapsed -evp aes-256-gcm
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-gcm for 3s on 16 size blocks: 20936402 aes-256-gcm's in 3.05s
Doing aes-256-gcm for 3s on 64 size blocks: 9676093 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 256 size blocks: 3293017 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 1024 size blocks: 907450 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 8192 size blocks: 117760 aes-256-gcm's in 3.04s
OpenSSL 1.0.2n  7 Dec 2017
built on: reproducible build, date unspecified
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: cc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -pthread -D_THREAD_SAFE -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -O3 -Wall -O2 -pipe  -fPIE -fPIC -Werror -Qunused-arguments -fstack-protector-all -fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-gcm     109661.77k   206423.32k   281004.12k   308938.41k   317430.10k

These values sound reasonable to me. Without the -evp flag I get ~30 MB/s, makes sense. However, it is not relevant if aesni.ko is loaded. After unloading it, the speeds stay exactly the same. This is probably due to OpenSSL using some custom implementation and not aesni.ko.

The OpenVPN config looks as follows:

dev ovpnc1
verb 1
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_client1.pid
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-256-GCM
auth SHA512
up /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkup
down /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkdown
tls-client
client
nobind
management /var/etc/openvpn/client1.sock unix
remote xxxxx
ca /var/etc/openvpn/client1.ca
cert /var/etc/openvpn/client1.cert
key /var/etc/openvpn/client1.key
resolv-retry infinite
compress

Now comes the curious part. An iperf3 through the VPN link from server to client gives 10-25 Mbit/s, even though the DSL line this goes through has 50 Mbit/s. iperf directly through WAN (without VPN) proves the 50 Mbit/s work. The CPU of the OPNsense host is at 25%, meaning one of the four cores is under full load.

Changing the cipher to AES-256-CBC gives roughly the same results.

I also tried connecting to the OpenVPN server directly from an i7 laptop behind the OPNsense box, reaching 48 Mbit/s without problems.

So apparently OpenVPN is not using the hardware crypto while OpenSSL does... Do you have an idea why this is the case?

Thanks in advance!

Best regards
Robert


EDIT: OPNsense version is as follows:
OPNsense 18.1.5-amd64
FreeBSD 11.1-RELEASE-p8
OpenSSL 1.0.2n 7 Dec 2017

[roschi]

Hi again,

I did some tuning of the KVM settings for the OPNsense guest. Specifically, I reduced the number of VCPUs from 4 to 2 and pinned them to cores 0 and 2 of the host. That seemed to help somehow, but the performance is still not how I'd expect it to be. I now reach 42 Mbit/s, but qemu-system-x86 still shows 145% cpu usage... actually it saturates one core completely.

Another quite shocking observation I made is that the performance of the whole VM networking seems to be abnormal. An iperf from the guest to the host, shows only 400-600 Mbit/s through the bridge with full cpu utilization. With UDP, it's only about 150-200 Mbit/s. I'd normally expect multiple Gbit/s from a bridged host/guest setup with virtio drivers. When using e1000 instead, I only get 200 Mbit/s with iperf3 in TCP mode, so that's even worse.

I then hooked up a grml live image to a second VM, also bridged with the host, and tried iperf between the host and that VM. Guess what? It has severe performance issues as well. With virtio I get 900 Mbit/s and with e1000 ~200 Mbit/s. CPU is always under full load during these tests. I've got actually no idea what's going on there, but it is not related to OPNsense at least.

Maybe one of you has an idea anyway...

Thanks!

Best regards
Robert

[elektroinside]

It doesn't matter how many cores you have, as the OpenVPN server instance is single threaded, meaning it will always use one cpu core. This is true with any OpenVPN implementation. If you have a good single-core performance, it should be fast enough, if not, it won't.

You could also follow this thread for better OpenVPN performance, even though it was written for something else (IDPS):
https://forum.opnsense.org/index.php?topic=6590.0

[roschi]

Hi,

Thanks for your reply.

I know that OpenVPN is single-threaded. But I expect more than 5MB/s on a CPU with 1,6/2,6 GHz and AES-NI support though. Consider that the OpenSSL speed benchmark showed that it's able to encrypt between 100 and 300 MB/s, even in the virtualized environment.

As I wrote, there seems to be something really weird with the KVM networking performance in my case. I just did a BIOS upgrade today, resetted everything back to factory defaults and went through every single setting to verify everything is configured correctly... But no difference so far.

Best regards
Robert

[roschi]

Ah, and in the thread you posted they write the tunables are only possible with igb NICs, but I'm using virtio drivers.

[elektroinside]

I'm just pointing out clues that may also help you.
For example, there may be some tunables out there available for virtio drivers, as they influence OpenVPN as well.
Also, considering your test results, indeed, OpenVPN performance should be better, but only CPU-wise, as there is no real network traffic included in those benchmarks.

But rest assured, OpenVPN will be using AES-NI whatever the setting in the OPNsense GUI if this CPU instruction set is available.

[roschi]

Ok, but what else if not cryptography could cause full cpu saturation at 5 MB/sec VPN throughput?

Regarding the virtio NICs, everybody suggests disabling hardware offloading of checksum calculation, which is the default behaviour of OPNsense and I double-checked it. Nevertheless there must be something wrong with the virtualization, since 500 MBit/s bridge throughput are simply not feasible. I'll try to figure out what that is, even though I've got no idea where to start.