Call for testers - new CPU microcode update plugins

[Patrick M. Hausen]

If you completely remove the ucode update plugin, you still cannot upgrade?

[OPNenthu]

Sorry, let me clarify:

I'm already on 26.7 because I had proactively removed the microcode plugin before upgrading from 26.1.  I took a snapshot after the OS upgrade and before installing the microcode plugin again.  Let's call this snapshot X.  From this starting point I then did the following.

Attempt 1: Re-installed the microcode plugin (intel) from the 26.7 UI
-> console freeze on reboot; problem confirmed

Rolled back to snapshot X.

Attempt 2: Installed the plugin & manually upgraded the bootcode by copying /boot/loader.efi to /boot/efi/efi/boot/bootx64.efi and /boot/efi/efi/freebsd/loader.efi
-> console freeze on reboot; no change

Rolled back to snapshot X.  Confirmed system was again booting normally.

Attempt 3: Followed the steps for Intel in post #1 only (no bootcode upgrade)
-> console freeze on reboot; no change

So I've again rolled back to snapshot X and this is where I'm sitting currently.

[OPNenthu]

Ok, I think this was my fault.  I re-did everything as below:

1. From a running 26.1.11_10 version, first remove the os-cpu-microcode-intel plugin
2. Upgrade to 26.7
3. Update bootloader as per https://forum.opnsense.org/index.php?topic=48145.msg243083#msg243083
4. Apply the steps in @Patrick's post #1 (for Intel)
5. Reboot

Now, I see the late loading in the console just before the initialization of interfaces and the output is not freezing!  It appears I also don't have any ucode updates available as nothing newer is getting applied, but that's OK.

Code Select Expand

Updating CPU Microcode...
CPU: Intel(R) Celeron(R) N5105 @ 2.00GHz (1996.80-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906c0  Family=0x6  Model=0x9c  Stepping=0
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2394a2c3<FSGSBASE,TSCADJ,FDPEXC,SMEP,ERMS,NFPUSG,PQE,RDSEED,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA>
  Structured Extended Features2=0x18400124<UMIP,WAITPKG,GFNI,RDPID,MOVDIRI,MOVDIR64B>
  Structured Extended Features3=0xfc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x14020c6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
Done.

So from this state now with the manually installed packages, what will I need to do on the next OPNsense update?  Will it automatically clear the manual packages and apply the updated plugin?

Thanks!

[Patrick M. Hausen]

With the updated bootloader you would not need my manual steps.

[OPNenthu]

I'm willing to retry it in case I messed up earlier but that is what I already tried in post #21.  You had asked if updating the bootloader alone would fix the issue for me but it didn't.

Maybe the order of operations matters.  I already had the plugin installed when I did the bootloader update.  Let me instead try the bootloader update without the ucode plugin, reboot, install the plugin, reboot again.  I'll report back shortly.

[OPNenthu]

Reproducible boot freeze:

1. Upgrade to 26.7 (without ucode plugin installed)
2. Update the bootloader
3. Reboot
4. Install the os-cpu-microcode-intel plugin from the GUI
5. Reboot
   -> console locks up on the "Loading splash ok" line.

Code Select Expand

   Autoboot in 0 seconds. [Space] to pause  26.7  ``The Road Ahead''   /
Loading kernel...
/boot/kernel/kernel text=0x18d968 text=0xe7b06c text=0x47b433 data=0x180+0xe80 data=0x1a84c0+0x857b40 0x8+0x1abe48+0x8+0x1d1c04
Loading configured modules...
/boot/kernel/pf.ko size 0xbd6e8 at 0x277e000
/boot/kernel/zfs.ko size 0x628648 at 0x283c000
/boot/kernel/pfsync.ko size 0x12340 at 0x2e65000
/boot/kernel/carp.ko size 0x11b40 at 0x2e78000
/boot/kernel/pflog.ko size 0x3c08 at 0x2e8a000
/boot/kernel/opensolaris.ko size 0x1e2f0 at 0x2e8e000
/etc/hostid size=0x25
/boot/kernel/if_gre.ko size 0xaa78 at 0x2ead000
/boot/kernel/if_lagg.ko size 0x15ef8 at 0x2eb8000
loading required module 'if_infiniband'
/boot/kernel/if_infiniband.ko size 0x3558 at 0x2ece000
/boot/kernel/if_enc.ko size 0x4be0 at 0x2ed2000
/boot/firmware/intel-ucode.bin size=0x1050000
/boot/entropy size=0x1000
/boot/kernel/if_bridge.ko size 0x10a68 at 0x3f28000
loading required module 'bridgestp'
/boot/kernel/bridgestp.ko size 0x8af8 at 0x3f39000
staging 0x6c000000 (not copying) tramp 0x76df6000 PT4 0x76ded000
Start @ 0xffffffff8038e000 ...
Loading splash ok

Working:

The original method in this thread.

1. Upgrade to 26.7 (without ucode plugin installed)
2. Install the Intel ucode packages manually from post #1
3. Reboot
  -> working

Code Select Expand

Generating configuration: templates...done
>>> Invoking early script 'cpu-microcode'
Updating CPU Microcode...
CPU: Intel(R) Celeron(R) N5105 @ 2.00GHz (1996.80-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906c0  Family=0x6  Model=0x9c  Stepping=0
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2394a2c3<FSGSBASE,TSCADJ,FDPEXC,SMEP,ERMS,NFPUSG,PQE,RDSEED,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA>
  Structured Extended Features2=0x18400124<UMIP,WAITPKG,GFNI,RDPID,MOVDIRI,MOVDIR64B>
  Structured Extended Features3=0xfc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x14020c6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
Done.

So, at least on my particular system the bootloader update is not enough to get the microcode plugin working.

The method in post #1 works and I probably did something wrong in my last try.

Hope someone else with Intel can validate this.

[Patrick M. Hausen]

Thanks a lot for testing again. How did you update the boot loader?

[OPNenthu]

Starting point:

Code Select Expand

root@firewall:~ # uname -a
FreeBSD firewall.h1.home.arpa 15.1-RELEASE-p1 FreeBSD 15.1-RELEASE-p1 stable/26.7-n283674-12334a596709 SMP amd64
root@firewall:~ # cat /etc/fstab
# Device        Mountpoint    FStype    Options        Dump    Pass#
/dev/gpt/efiboot0    /boot/efi    msdosfs    rw,noauto    2    2
/dev/nda0p3        none        swap    sw        0    0
root@firewall:~ # gpart show
=>       40  488397088  nda0  GPT  (233G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  471085056     4  freebsd-zfs  (225G)
  488396800        328        - free -  (164K)

root@firewall:~ # ls -l /boot/efi
total 0

Procedure:

Code Select Expand

root@firewall:~ # mkdir -p /boot/efi/efi/boot /boot/efi/efi/freebsd
root@firewall:~ # cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi
root@firewall:~ # cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
root@firewall:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda0
partcode written to nda0p2
bootcode written to nda0
root@firewall:~ #

Reference: https://forum.opnsense.org/index.php?topic=48145.msg243083#msg243083

[Patrick M. Hausen]

Did you mount /boot/efi, first? From the output of your "ls" command I fear you did not. So possibly you copied the boot loader to a directory named /boot/efi on the root filesystem.

[OPNenthu]

Ah, that's what I missed.  Thanks!

Trying again...

[newsense]

I've updated 4 FWs yesterday using the latest loaders-update script v1.4.0 (last time I've tested it was 1.3.2).

2xFW4c and a DEC750 on ZFS and an APU4 on UFS.

Unfortunately neither of these FWs appear to be in scope for a microcode update on boot. The bootloader update though seems to be required on major OS upgrades so that's one thing less to worry about.

[OPNenthu]

Did you mount /boot/efi, first? From the output of your "ls" command I fear you did not. So possibly you copied the boot loader to a directory named /boot/efi on the root filesystem.

Yes, this was the issue.  I had put the "noauto" mount option on the efi boot partition some time ago and didn't mount it manually.

All good now.  The bootloader is updated and the os-cpu-microcode-intel plugin is installed and loading early:

Code Select Expand

---<<BOOT>>---
Copyright (c) 1992-2025 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 15.1-RELEASE-p1 stable/26.7-n283674-12334a596709 SMP amd64
FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2)
VT(vga): resolution 640x480
CPU microcode: no matching update found
CPU: Intel(R) Celeron(R) N5105 @ 2.00GHz (1996.80-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906c0  Family=0x6  Model=0x9c  Stepping=0
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2394a2c3<FSGSBASE,TSCADJ,FDPEXC,SMEP,ERMS,NFPUSG,PQE,RDSEED,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA>
  Structured Extended Features2=0x18400124<UMIP,WAITPKG,GFNI,RDPID,MOVDIRI,MOVDIR64B>
  Structured Extended Features3=0xfc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x14020c6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics

Thanks again, Patrick

[opn_mndr12101]

I just ran into that microcode issue. I have two (almost) identical devices, TK Edge 4L and TK Edge4Go¹, both Intel Celeron J3455. The Edge 4L is affected by this issue, TK Edge4Go is not. Should I remove os-cpu-microcode-intel from the working Edge4Go, too?
Why is one affected, the other not?


1) https://www.thomas-krenn.com/de/produkte/low-energy-systeme

Code Select Expand

kenv smbios.bios.vendor: American Megatrends Inc./Advantech
kenv smbios.bios.version: T011G206
kenv smbios.bios.reldate: 04/18/2023
sysctl hw.model: hw.model: Intel(R) Celeron(R) CPU J3455 @ 1.50GHz

[meyergru]

Because it is a complex timing issue that manifests itself only under specific circumstances.

[Patrick M. Hausen]

@opn_mndr12101 Update the boot loaders on both devices and you can keep the plugin.