OPNsense Forum
English Forums => Hardware and Performance => Topic started by: framura on March 09, 2016, 11:57:41 pm
-
Hi,
I tried to verify openssl performance on my OPNSense machine (4 vCPUs on ESXi 6): I compared these results with those obtained on a Ubuntu 15 server machine (2 vCPUs on same host).
I run this command on Ubuntu and OPNSense:
openssl speed -evp aes-128-cbc
and on OPNSense also
openssl speed -evp aes-128-cbc -engine cryptodev
Output obtained shows me no difference between these two commands on OPNSense and a huge difference with Ubuntu:
OPNSense (no cryptodev)
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 46929.27k 165008.10k 662520.67k 3014012.56k 30838620.16k
OPNSEnse (cryptodev)
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 45672.01k 183696.55k 576098.68k 2877417.92k 20640869.03k
Ubuntu 15
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 209778.60k 321537.47k 386955.43k 406307.21k 413696.00k
Do you have any idea?
Thanks
-
Can you rerun these with the following binary? Due to the way FreeBSD is built we have two versions and the console defaults to the wrong one...
# /usr/local/bin/openssl
-
Here's my dump with base OpenSSL:
root@sensey:~ # openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 620655 aes-128-cbc's in 0.42s
Doing aes-128-cbc for 3s on 64 size blocks: 605377 aes-128-cbc's in 0.33s
Doing aes-128-cbc for 3s on 256 size blocks: 551208 aes-128-cbc's in 0.37s
Doing aes-128-cbc for 3s on 1024 size blocks: 405373 aes-128-cbc's in 0.22s
Doing aes-128-cbc for 3s on 8192 size blocks: 120284 aes-128-cbc's in 0.07s
OpenSSL 1.0.1p-freebsd 9 Jul 2015
built on: date not available
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 23538.92k 118077.34k 384297.53k 1897608.92k 14014101.73k
And this is LibreSSL (the OpenSSL flavour should be equivalent in numbers), which is used by our code:
root@sensey:~ # /usr/local/bin/openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 42196792 aes-128-cbc's in 2.99s
Doing aes-128-cbc for 3s on 64 size blocks: 15461014 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 4929140 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 1316453 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 167027 aes-128-cbc's in 3.00s
LibreSSL 2.2.6
built on: date not available
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 225637.15k 329834.97k 420619.95k 449349.29k 456095.06k
-
@franco - It's interesting it seems LibreSSL is not using cryptodev, which is a good thing.
@framura - If you want the best openSSL performance, you need to build a custom kernel and remove cryptodev from https://github.com/opnsense/tools/blob/master/config/16.1/SMP. See https://github.com/opnsense/tools/issues/26 for more details.
Also, when posting results, paste the whole thing. In Franco's test using base OpenSSL, we can see that the test is not accurate as it doesn't run for 3s. Looking at your numbers, I expect the same, because they're unrealistic. You don't encrypt at 30GB/s. The numbers on Ubuntu are correct. Add "-elapsed" to the command line if you want the real numbers on OPNsense with cryptodev loaded.
Also, you don't need to add "-engine cryptodev", evp is already using cryptodev if present and that's what's slowing thing down.
-
Yes, cryptodev engine support was kicked from LibreSSL very early on. :)
-
Thanks both, @franco and @interfaSys, for your replies.
So, if I undestand correctly, OPNSense loads aesni.ko (if I set into settings preferences) but openssl (and openvpn) use /dev/crypto always (if device is present), also if I don't set it in openvpn configuration: when both aesni module and crypto device are present, openssl (and openvpn) slows down.
So, possible solutions (at this moment):
1) Build a custom kernel, like @interfaSys says
2) Switch to LibreSSL flavour: but LibreSSL uses AES-NI instructions? From a another thread, I understood LibreSSL is not able to use them (or am I wrong?)
Thanks again
P.S.: I found also this: https://calomel.org/aesni_ssl_performance.html
-
No, I think AES-NI is implemented in (envelope) assembler code directly in LibreSSL and OpenSSL. It does not need anything other than AES-NI instructions from the hardware, unless I don't understand how it works.
-
@franco is correct. OpenSSL comes with its own implementation in its evp engine and it doesn't require any module to be loaded.
The only reason you'd want to load aesni.ko is if you have other kernel components which can use it to accelerate encryption. I was told IPsec uses it per example. But you don't need to load cryptodev as that's used as a bridge to give userland access to crypto accelerators kernel modules (if you have a PCI-X card per example).
OpenVPN uses OpenSSL's evp engine, but in my tests it shows zero gain in terms of speed when using cryptodev or not. It just spends more time in the kernel space if cryptodev is loaded.
And careful, crypto is not cryptodev ;). You need to load crypto, not cryptodev.
-
Just tried to use LibreSSL: I changed flavour in System:Settings:General but
if I run "openssl version" I get
OpenSSL 1.0.1p-freebsd 9 Jul 2015
and if I run "/usr/local/bin/openssl version" I get
OpenSSL 1.0.2g 1 Mar 2016
In franco's run I see "LibreSSL 2.2.6".
-
Sorry,
reply to myself: I changed ssl flavour but I did't realise I must also to update some packages.
Now I updated my system and finally I get LibreSSL 2.2.6: sorry, my mistake.
Now I will try openvpn performance and CPU usage.
-
I report some results.
I made some speed test with and without OPNSense, with and without OpenVPN and with different cypher (AES-256-CBC and BF-CBC, 128bit).
Without OPNSense and without VPN, I maxed out my Internet bandwidth (100Mbps).
Without OPNSense and with VPN (directly from my desktop machine, I7), I get 80-90% Internet bandwidth (no difference between AES and BF).
With OPNSense and without VPN, I maxed out my Internet bandwidth and with 8-10% CPU on OPNSense machine
With OPNSense and with VPN, I get 80-90% Internet bandwidth (with AES-256) and with 25-30% CPU.
With OPNSense and with VPN, I get 70-80% Internet bandwidth (with BF-CBC) and with 25-30% CPU.
What do you think?
Thanks
-
Blowfish looks a little off, but that might be FreeBSD. Top 10% can be lost in protocol overhead. With that in mind, these numbers are solid. :)
Regarding firmware flavours: This tends to happen with users, we are already planning to rework the firmware parts to make it clearer that an update must be run in order to finish the transition. Sorry about that.
-
I agree, these numbers are good.
When with openvpn will be possible to use aes-256-gcm, perhaps they will become even better.
-
When with openvpn will be possible to use aes-256-gcm, perhaps they will become even better.
Not sure...
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 157757.68k 213745.17k 244716.20k 253700.10k 257092.77k
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-gcm 97815.09k 179295.42k 236719.87k 259856.57k 266351.96k