PCEngines APU2/APU3/APU4 running on 20.7

[amichel]

Hi,
just an idea here:

Any suggestions on how to debug this better?  Can I reset the WAN DHCP from the console so I can watch for errors?  It's driving me nuts since a reboot tends to fix up the problem right away.

Why don't you use monit to check for the connection?
You could for example create a connection test to any IP available from the firewall and in case it is not reachable you could initiate a reload afer let's say 5 missed attemts and initiate a reboot afer 15 attempts.
Thats the way I monitor my internet connection.
amichelf

[almodovaris]

I have an APU2 4 GB. I always update to latest mainline BIOS and latest Opnsense, at most with one week delay. I have Ziggo 500 down 40 up. The APU2 is in the DMZ of my modem/router.

I have installed Sensei. No big problems that I could report.

[l8gravely]

I just recently upgraded my old APU1 running 20.7.<recent> to an APU4d4 because I keep getting drops of my WAN connection, it's like the DCHP times out and the router can't renew it properly.  I'm working along and suddenly things freeze and I can't work, or my kids come yelling that the Wifi is down.  Heh. 

So I goto Interfaces -> Overview -> igb3 and hit the 'renew' button.  Which usually works, but not always.  I might have to hit it multiple times. 

I log via syslog to my main home system which is on 24x7, so can anyone give good suggestions for what to look for in the logs to help diagnose this problem?  It's really frustrating when the kids are in school, or us parent's are on conference calls. 

Here's a snippet from my log:

opnsense[32236]: /usr/local/etc/rc.configure_interface: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dhclient.igb3.pid' 'igb3'' returned exit code '15', the output was 'dhclient 41052 - - Starting delete_old_states() dhclient 44642 - - Comparing IPs: Old: 66.189.75.104 New: dhclient 47831 - - Removing states from old IP '66.189.75.104' (new IP '') 0 states cleared killed 0 src nodes from 1 sources and 0 destinations DHCPREQUEST on igb3 to 255.255.255.255 port 67 DHCPREQUEST on igb3 to 255.255.255.255 port 67 DHCPREQUEST on igb3 to 255.255.255.255 port 67 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 2 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 3 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 3 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 6 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 8 DHCPDISCOVER on igb3 to 255.255.255.255 port 67 interval 19'
Dec  1 11:50:30 gw dhclient[3571]: Starting delete_old_states()

It looks like it's a failure to properly renew the WAN DHCP client IP address for some reason.  Has anyone else seen this issue?  Any suggestions on what I can do to make it work better?

[almodovaris]

Yup, give it a static lease. That means no bridge mode for your modem, use DMZ.

[Ricardo]

Has anyone experienced lower than expected AES / VPN troughput recently?

I did a measurement on my APU2, and it performs significantly worse, than 2 years earlier:

openssl speed -elapsed -evp aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 11056362 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 5231233 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 1850707 aes-128-cbc's in 3.05s
Doing aes-128-cbc for 3s on 1024 size blocks: 505589 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 64769 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 32500 aes-128-cbc's in 3.01s
OpenSSL 1.1.1d-freebsd  10 Sep 2019
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc      58967.26k   111599.64k   155497.35k   172574.38k   176862.55k   177032.31k

--> approx. 172Mbit/sec traffic using 1024bytes blocks (measurement repeated multiple times, router is lightly loaded, 80+% idle)


Earlier (Opnsense 18.x, OpenSSL 1.0.2k-freebsd 26 Jan 2017): it was higher:

openssl speed -elapsed -evp aes-128-cbc

type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 109850.66k 157501.93k 194130.64k 205928.11k 210187.46k

--> around 205-210 Mbit/sec

Thats quite a big steady 20% degradation.

[franco]

The correct binary to check is (and always was) /usr/local/bin/openssl


Cheers,
Franco

[Ricardo]

The correct binary to check is (and always was) /usr/local/bin/openssl


Cheers,
Franco

Did you spot any mistake?

[hushcoden]

This is what I get with my apu2e4:

Code Select Expand

root@hush:~ # openssl speed -elapsed -evp aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 15080231 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 7226969 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 2525328 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 570466 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 73644 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 36643 aes-128-cbc's in 3.00s
OpenSSL 1.1.1d-freebsd  10 Sep 2019
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc      80427.90k   154175.34k   215494.66k   194719.06k   201097.22k   200119.64k


[franco]

Guys, for the love of the Gods please use /usr/local/bin/openssl ;)

# which openssl
/usr/bin/openssl


Cheers,
Franco

[hushcoden]

Slightly better:

Code Select Expand

root@hush:~ # /usr/local/bin/openssl speed -elapsed -evp aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 11289033 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 5595084 aes-128-cbc's in 3.01s
Doing aes-128-cbc for 3s on 256 size blocks: 2235362 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 690092 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 89374 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 44699 aes-128-cbc's in 3.00s
OpenSSL 1.1.1h  22 Sep 2020
built on: Tue Oct 20 22:46:58 2020 UTC
options:bn(64,64) rc4(8x,int) des(int) aes(partial) blowfish(ptr)
compiler: cc -fPIC -pthread -Wa,--noexecstack -Qunused-arguments -O2 -pipe  -DHARDENEDBSD -fPIE -fPIC -fstack-protector-all -fno-strict-aliasing -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -D_THREAD_SAFE -D_REENTRANT -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc      60208.18k   119051.76k   190750.89k   235551.40k   244050.60k   244116.14k

[Ricardo]

Guys, for the love of the Gods please use /usr/local/bin/openssl ;)

# which openssl
/usr/bin/openssl


Cheers,
Franco

1) For the love of god, why do you guys deploy 2 different openssl versions on the same opnsense without describing this trapmine?

2) In my case, it didnt give better results:

~ # /usr/local/bin/openssl speed -elapsed -evp aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 10431991 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 5030317 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 1792666 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 505478 aes-128-cbc's in 3.02s
Doing aes-128-cbc for 3s on 8192 size blocks: 64928 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 32502 aes-128-cbc's in 3.00s
OpenSSL 1.1.1h  22 Sep 2020
built on: Tue Oct 20 22:46:58 2020 UTC
options:bn(64,64) rc4(8x,int) des(int) aes(partial) blowfish(ptr)
compiler: cc -fPIC -pthread -Wa,--noexecstack -Qunused-arguments -O2 -pipe  -DHARDENEDBSD -fPIE -fPIC -fstack-protector-all -fno-strict-aliasing -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -D_THREAD_SAFE -D_REENTRANT -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc      55637.29k   107313.43k   152974.17k   171642.52k   177296.73k   177504.26k

[almodovaris]

Well, you should know that APUs are now considered low-end hardware. So be content with that it offers. For the home user (hobbyist) it is a perfect box. But it is not suitable for very demanding tasks.

[Gary7]

FWIW, I ran the same command on my APU2D4
~ # /usr/local/bin/openssl speed -elapsed -evp aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 16220482 aes-128-cbc's in 3.01s
Doing aes-128-cbc for 3s on 64 size blocks: 7941393 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 2918949 aes-128-cbc's in 3.02s
Doing aes-128-cbc for 3s on 1024 size blocks: 816008 aes-128-cbc's in 3.02s
Doing aes-128-cbc for 3s on 8192 size blocks: 105686 aes-128-cbc's in 3.02s
Doing aes-128-cbc for 3s on 16384 size blocks: 52871 aes-128-cbc's in 3.02s
OpenSSL 1.1.1h  22 Sep 2020
built on: Tue Oct 20 22:46:58 2020 UTC
options:bn(64,64) rc4(8x,int) des(int) aes(partial) blowfish(ptr)
compiler: cc -fPIC -pthread -Wa,--noexecstack -Qunused-arguments -O2 -pipe  -DHARDENEDBSD -fPIE -fPIC -fstack-protector-all -fno-strict-aliasing -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -D_THREAD_SAFE -D_REENTRANT -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc      86284.54k   169416.38k   247793.06k   277087.57k   286356.08k   286507.81k

Coreboot:  v4.13.0.1
Core Performance Boost - Currently Enabled
I'm the only one on the system so I set:
vm.pmap.pti = 0
hw.ibrs_disable = 0
among some other optimization settings

[Gary7]

I don't remember when Meltdown mitigation and Spectre V2 mitigation was introduced.
We know that mitigating these issues in the O/S causes a performance "hit". I just don't know how much.

My APU2D4 is for my home network and all inbound traffic is disabled. So, disabling the mitigation was an acceptable risk for me in order to get best performance.

According to AMD, their CPUs are not vulnerable to the Meltdown vulnerability. So, it should be safe to disable Meltdown mitigation, vm.pmap.pti = 0

[almodovaris]

Yup, on my APU2C4:

/usr/local/bin/openssl speed -elapsed -evp aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 13947948 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 7114424 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 2589608 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 733300 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 93373 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 45467 aes-128-cbc's in 3.00s
OpenSSL 1.1.1h  22 Sep 2020
built on: Tue Oct 20 22:46:58 2020 UTC
options:bn(64,64) rc4(8x,int) des(int) aes(partial) blowfish(ptr)
compiler: cc -fPIC -pthread -Wa,--noexecstack -Qunused-arguments -O2 -pipe  -DHARDENEDBSD -fPIE -fPIC -fstack-protector-all -fno-strict-aliasing -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -D_THREAD_SAFE -D_REENTRANT -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc      74389.06k   151774.38k   220979.88k   250299.73k   254970.54k   248310.44k