Openssl performance

[framura]

Hi,

I tried to verify openssl performance on my OPNSense machine (4 vCPUs on ESXi 6): I compared these results with those obtained on a Ubuntu 15 server machine (2 vCPUs on same host).

I run this command on Ubuntu and OPNSense:

openssl speed -evp aes-128-cbc

and on OPNSense also

openssl speed -evp aes-128-cbc -engine cryptodev

Output obtained shows me no difference between these two commands on OPNSense and a huge difference with Ubuntu:

OPNSense (no cryptodev)
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes        256 bytes       1024 bytes      8192 bytes
aes-128-cbc  46929.27k  165008.10k   662520.67k   3014012.56k   30838620.16k


OPNSEnse (cryptodev)
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes        256 bytes       1024 bytes      8192 bytes
aes-128-cbc  45672.01k  183696.55k   576098.68k   2877417.92k   20640869.03k


Ubuntu 15

The 'numbers' are in 1000s of bytes per second processed.
type                16 bytes        64 bytes       256 bytes      1024 bytes    8192 bytes
aes-128-cbc     209778.60k   321537.47k   386955.43k   406307.21k   413696.00k


Do you have any idea?

Thanks
 

[franco]

Can you rerun these with the following binary? Due to the way FreeBSD is built we have two versions and the console defaults to the wrong one...

# /usr/local/bin/openssl

[franco]

Here's my dump with base OpenSSL:

Code: [Select]

root@sensey:~ # openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 620655 aes-128-cbc's in 0.42s
Doing aes-128-cbc for 3s on 64 size blocks: 605377 aes-128-cbc's in 0.33s
Doing aes-128-cbc for 3s on 256 size blocks: 551208 aes-128-cbc's in 0.37s
Doing aes-128-cbc for 3s on 1024 size blocks: 405373 aes-128-cbc's in 0.22s
Doing aes-128-cbc for 3s on 8192 size blocks: 120284 aes-128-cbc's in 0.07s
OpenSSL 1.0.1p-freebsd 9 Jul 2015
built on: date not available
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      23538.92k   118077.34k   384297.53k  1897608.92k 14014101.73k
And this is LibreSSL (the OpenSSL flavour should be equivalent in numbers), which is used by our code:

Code: [Select]

root@sensey:~ # /usr/local/bin/openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 42196792 aes-128-cbc's in 2.99s
Doing aes-128-cbc for 3s on 64 size blocks: 15461014 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 4929140 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 1316453 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 167027 aes-128-cbc's in 3.00s
LibreSSL 2.2.6
built on: date not available
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc     225637.15k   329834.97k   420619.95k   449349.29k   456095.06k

[interfaSys]

@franco - It's interesting it seems LibreSSL is not using cryptodev, which is a good thing.

@framura - If you want the best openSSL performance, you need to build a custom kernel and remove cryptodev from https://github.com/opnsense/tools/blob/master/config/16.1/SMP. See https://github.com/opnsense/tools/issues/26 for more details.
Also, when posting results, paste the whole thing. In Franco's test using base OpenSSL, we can see that the test is not accurate as it doesn't run for 3s. Looking at your numbers, I expect the same, because they're unrealistic. You don't encrypt at 30GB/s. The numbers on Ubuntu are correct. Add "-elapsed" to the command line if you want the real numbers on OPNsense with cryptodev loaded.
Also, you don't need to add "-engine cryptodev", evp is already using cryptodev if present and that's what's slowing thing down.

[franco]

Yes, cryptodev engine support was kicked from LibreSSL very early on.

[framura]

Thanks both, @franco and @interfaSys, for your replies.

So, if I undestand correctly, OPNSense loads aesni.ko (if I set into settings preferences) but openssl (and openvpn) use /dev/crypto always (if device is present), also if I don't set it in openvpn configuration: when both aesni module and crypto device are present, openssl (and openvpn) slows down.

So, possible solutions (at this moment):

1) Build a custom kernel, like @interfaSys says
2) Switch to LibreSSL flavour: but LibreSSL uses AES-NI instructions? From a another thread, I understood LibreSSL is not able to use them (or am I wrong?)

Thanks again

P.S.: I found also this: https://calomel.org/aesni_ssl_performance.html

[franco]

No, I think AES-NI is implemented in (envelope) assembler code directly in LibreSSL and OpenSSL. It does not need anything other than AES-NI instructions from the hardware, unless I don't understand how it works.

[interfaSys]

@franco is correct. OpenSSL comes with its own implementation in its evp engine and it doesn't require any module to be loaded.

The only reason you'd want to load aesni.ko is if you have other kernel components which can use it to accelerate encryption. I was told IPsec uses it per example. But you don't need to load cryptodev as that's used as a bridge to give userland access to crypto accelerators kernel modules (if you have a PCI-X card per example).

OpenVPN uses OpenSSL's evp engine, but in my tests it shows zero gain in terms of speed when using cryptodev or not. It just spends more time in the kernel space if cryptodev is loaded.

And careful, crypto is not cryptodev . You need to load crypto, not cryptodev.

[framura]

Just tried to use LibreSSL: I changed flavour in System:Settings:General but

if I run "openssl version" I get

OpenSSL 1.0.1p-freebsd 9 Jul 2015

and if I run "/usr/local/bin/openssl version" I get

OpenSSL 1.0.2g  1 Mar 2016

In franco's run I see "LibreSSL 2.2.6".

[framura]

Sorry,

reply to myself: I changed ssl flavour but I did't realise I must also to update some packages.

Now I updated my system and finally I get LibreSSL 2.2.6: sorry, my mistake.

Now I will try openvpn performance and CPU usage.

[framura]

I report some results.

I made some speed test with and without OPNSense, with and without OpenVPN and with different cypher (AES-256-CBC and BF-CBC, 128bit).

Without OPNSense and without VPN, I maxed out my Internet bandwidth (100Mbps).
Without OPNSense and with VPN (directly from my desktop machine, I7), I get 80-90% Internet bandwidth (no difference between AES and BF).
With OPNSense and without VPN, I maxed out my Internet bandwidth and with 8-10% CPU on OPNSense machine
With OPNSense and with VPN, I get 80-90% Internet bandwidth (with AES-256) and with 25-30% CPU.
With OPNSense and with VPN, I get 70-80% Internet bandwidth (with BF-CBC) and with 25-30% CPU.

What do you think?

Thanks

[franco]

Blowfish looks a little off, but that might be FreeBSD. Top 10% can be lost in protocol overhead. With that in mind, these numbers are solid.

Regarding firmware flavours: This tends to happen with users, we are already planning to rework the firmware parts to make it clearer that an update must be run in order to finish the transition. Sorry about that.

[framura]

I agree, these numbers are good.

When with openvpn will be possible to use aes-256-gcm, perhaps they will become even better.

[interfaSys]

When with openvpn will be possible to use aes-256-gcm, perhaps they will become even better.

Not sure...

Code: [Select]

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     157757.68k   213745.17k   244716.20k   253700.10k   257092.77k

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-gcm      97815.09k   179295.42k   236719.87k   259856.57k   266351.96k