Messages

Unicast vs. multicast seems to make no difference.

What makes a difference however ist disabling multiqueue in proxmox. Removing the queues=X parameter completely mitigates the issue.

However, I know of two other OPNsense on Proxmox installations that do not have this issue and run fine with queues=8.

Mystery.

Hi,

I am seeing the following issue:

"longer" tcp connections stall every one in n-th try.

I can reproduce this by running a while loop on the firewall itself that uses curl to get a 500mb file.
When the current download rate slowly drops to 0 and never recovers i have reproduced the issue.

All devices "behind" this setup are affected, larger downloads sometimes fail, docker image pulls have high chance of failure.

When I switch off pfsync the issue is resolved.

The firewall rule on the sync interface allows all traffic.

Pfsync is configured according to https://docs.opnsense.org/manual/how-tos/carp.html

a) can anybody reproduce?
b) is this a bug?

Martin

Update: I can reproduce this on two freshly installed 24.7.8 firewalls. Running the curl loop on both at the same time leads to stalls rather quickly.

Update2: I setup the same on two pfsense 2.7.2 firewalls. This does not reproduce the issue.

Same issue here after the 22.1 upgrade. OPNsense VMs hang on shutdown. After hard stop/start they then boot/run normally.

seing the same on multiple installations.

seems configd from before the upgrade to 21.1.9 is stuck:

Code Select Expand

45073  -  R      507:46.32 /usr/local/bin/python3 /usr/local/opnsense/service/configd_ctl.py -e -t 0.5 system even

after reboot it's gone.

Code Select Expand

service configd restart does not stop the old process.

ok, just to answer my own question:

backup old microcode
copy new microcode to /usr/local/share/cpucontrol/microcode_amd_fam16h.bin

Code Select Expand

pkg install devcpu-data
echo 'microcode_update_enable="YES"' >>/etc/rc.conf
service microcode_update start
cpucontrol -v -u /dev/cpuctl0
cpucontrol -v -u /dev/cpuctl1
cpucontrol -v -u /dev/cpuctl2
cpucontrol -v -u /dev/cpuctl3


There actually is new microcode in the wild, which applies successfully to the apu2.

https://github.com/platomav/CPUMicrocodes/blob/master/AMD/cpu00730F01_ver07030106_2018-02-09_88EDFAA0.bin

I have so far managed to load the microcode in my debian installation.

How would I load this in OPNsense?

Code Select Expand

[435975.155078] platform microcode: firmware: direct-loading firmware amd-ucode/microcode_amd_fam16h.bin
[435975.167741] microcode: CPU0: new patch_level=0x07030106
[435975.176174] microcode: CPU1: new patch_level=0x07030106
[435975.184785] microcode: CPU2: new patch_level=0x07030106
[435975.193171] microcode: CPU3: new patch_level=0x07030106


The spectre-meltdown-checker reflects that new microcode accordingly:

Code Select Expand


Spectre and Meltdown mitigation detection tool v0.35                                                                                                                   
                                                                                                                                                                       
Checking for vulnerabilities on current system                                                                                                                         
Kernel is Linux 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64                                                                                       
CPU is AMD GX-412TC SOC                                                                                                                                               
                                                                                                                                                                       
Hardware check                                                                                                                                                         
* Hardware support (CPU microcode) for mitigation techniques                                                                                                           
  * Indirect Branch Restricted Speculation (IBRS)                                                                                                                     
    * SPEC_CTRL MSR is available:  NO                                                                                                                                 
    * CPU indicates IBRS capability:  NO                                                                                                                               
  * Indirect Branch Prediction Barrier (IBPB)                                                                                                                         
    * PRED_CMD MSR is available:  YES                                                                                                                                 
    * CPU indicates IBPB capability:  YES  (IBPB_SUPPORT feature bit)                                                                                                 
  * Single Thread Indirect Branch Predictors (STIBP)                                                                                                                   
    * SPEC_CTRL MSR is available:  NO                                                                                                                                 
    * CPU indicates STIBP capability:  NO                                                                                                                             
  * Enhanced IBRS (IBRS_ALL)                                                                                                                                           
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO                                                                                                           
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO                                                                                                       
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO):  NO                                                                                           
  * CPU microcode is known to cause stability problems:  NO                                                                                                           
* CPU vulnerability to the three speculative execution attacks variants                                                                                               
  * Vulnerable to Variant 1:  YES                                                                                                                                     
  * Vulnerable to Variant 2:  YES                                                                                                                                     
  * Vulnerable to Variant 3:  NO                                                                                                                                       
                                                                                                                                                                       
CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'                                                                                                           
* Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active)
* Kernel has array_index_mask_nospec:  YES  (1 occurence(s) found of 64 bits array_index_mask_nospec())
* Kernel has the Red Hat/Ubuntu patch:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active)
* Mitigation 1
  * Kernel is compiled with IBRS/IBPB support:  NO
  * Currently enabled features
    * IBRS enabled for Kernel space:  NO
    * IBRS enabled for User space:  NO
    * IBPB enabled:  NO
* Mitigation 2
  * Kernel compiled with retpoline option:  YES
  * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
> STATUS:  NOT VULNERABLE  (Mitigation: Full AMD retpoline)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface:  YES  (kernel confirms that your CPU is unaffected)
* Kernel supports Page Table Isolation (PTI):  YES
* PTI enabled and active:  NO
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (your CPU vendor reported your CPU model as not vulnerable)


Try an older BIOS Version. For me 4.5.5 or older did the trick.

I am comiling anyhow to have the default console output on COM3. Background: I am trying to build a dual APU2 box with COM3/COM4 internally crossconnected. COM1 is then free to connect switches or other serial stuff.

Other than that i think the occasional USB errors are gone with 4.6.6

I mitigated the reboot hang problem with a simple usb-hid based watchdog [1] - my apu2 with self-compiled 4.6.6 booted reliably ~600 times over night - with the help of that watchdog.


[1] https://www.ebay.de/itm/Interne-USB-Watchdog-Reset-Controller-PC-Stick-Crash-Blue-Screen-automatisch/263490474653

I have also managed to compile the recent BIOS versions 4.6.5 and 4.6.6.

I have 2 issues with these:

a) memtest does beep when it is entered, but there is no console output - have you tested the memtest feature in 4.6.5 or 4.6.6? Did I miss something when I compiled this?

b) the apu2 does sporadically hang during reboot - sometimes the first time, sometimes it takes 20 reboots to hang. The original 4.6.1 also shows this behaviour. 4.5.5 seems to reboot reliably. I have contacted pcengines' support for this. I am interested in your experience regarding this issue.

Cheers

Martin