Call for testers - new CPU microcode update plugins

[meyergru]

@(MARLOO):

So I understand what you are essentially answering to this "call for testing": "No, thank you, I am not going to test the potential fix, but rather wait for other people's confirmation that the fix works and only then apply it with the next hotfix."?

How clever. (Sorry for being a bit harsh)

P.S.: The whole point of this thread is to gain knowledge if the presumed fix works for affected people, so just you know.... the only reason I did not try is that I do not experience the problem myself even with early microcode loading.

[OPNenthu]

I just re-installed the microcode plugin after upgrade to 26.7 and mine looks affected.  Console getting stuck here:

Code Select Expand

   Autoboot in 0 seconds. [Space] to pause  26.7  ``The Road Ahead''   /
Loading kernel...
/boot/kernel/kernel text=0x18d968 text=0xe7b06c text=0x47b433 data=0x180+0xe80 data=0x1a84c0+0x857b40 0x8+0x1abe48+0x8+0x1d1c04
Loading configured modules...
/boot/kernel/zfs.ko size 0x628648 at 0x277e000
/boot/kernel/if_enc.ko size 0x4be0 at 0x2da7000
/boot/firmware/intel-ucode.bin size=0x1050000
/boot/kernel/if_lagg.ko size 0x15ef8 at 0x3dfc000
loading required module 'if_infiniband'
/boot/kernel/if_infiniband.ko size 0x3558 at 0x3e12000
/boot/kernel/carp.ko size 0x11b40 at 0x3e16000
/boot/kernel/if_gre.ko size 0xaa78 at 0x3e28000
/boot/kernel/pflog.ko size 0x3c08 at 0x3e33000
loading required module 'pf'
/boot/kernel/pf.ko size 0xbd6e8 at 0x3e37000
/boot/kernel/opensolaris.ko size 0x1e2f0 at 0x3ef5000
/boot/kernel/pfsync.ko size 0x12340 at 0x3f14000
/boot/entropy size=0x1000
/boot/kernel/if_bridge.ko size 0x10a68 at 0x3f28000
loading required module 'bridgestp'
/boot/kernel/bridgestp.ko size 0x8af8 at 0x3f39000
/etc/hostid size=0x25
staging 0x6c000000 (not copying) tramp 0x76df6000 PT4 0x76ded000
Start @ 0xffffffff8038e000 ...
Loading splash ok

To try the late loading fix, I think I need to first apply both of @franco's patches from post #11 and I need to upgrade my bootloader to the latest as per the comment in the commit.

Only then should I do the steps @Patrick wrote in post #1.  Is that right?

Thanks!

For reference: Protectli V1410, N5105, coreboot v0.9.4.

[Patrick M. Hausen]

@meyergru There seems to be a connection to an outdated boot loader causing this issue when loading early.

The whole ZFS pool upgrade and boot loader upgrade situation is a bit frustrating for the whole FreeBSD ecosystem, not only OPNsense. We really need a conclusive solution in upstream.

One of the things I am taking with me to Brussels for the upcoming FreeBSD developer summit.

[Patrick M. Hausen]

@OPNenthu

If you apply the steps outlined by me you do not need a boot loader update. That's the point. Similarly the changes by Franco that will be in 26.7.1 address exactly this issue for people without a current boot loader.

It would be interesting to see if a boot loader upgrade alone fixes things for you!

[(MARLOO)]

@ meyergru

Hi meyergru,

I see how my previous message could sound like "I'll wait for others to test first", and I understand why that comes across as not really helping the thread. That wasn't my intent.

To be clear: on my N5105 box with 4× i226‑V I did hit a concrete problem with the microcode plugin. After updating the i226 NVM to 2.32 and running OPNsense 26.1.4, with os-intel-microcode installed I was getting boot hangs whenever both WAN and LAN were connected (link stayed amber, system seemed stuck, no progress on console). I documented this here in the i226 thread (posts #198–#199) and the issue disappeared after I removed os-intel-microcode.

https://forum.opnsense.org/index.php?topic=48695.msg263682#msg263682

Because of that experience, I'm a bit cautious about re‑introducing any microcode plugin on this primary firewall. I'm absolutely willing to help verify the "late load" fix, but I'm trying to balance testing with not breaking a production device again.

OPNenthu  with very similar hardware (N5105, 4× i226‑V) is now reporting boot stalls after reinstalling the microcode plugin on 26.7. This matches what I experienced on 26.1.4, which is why I'm being cautious about re‑enabling microcode on this box until the "late load" fix is fully validated.

My intention isn't to avoid testing, just to make sure I do it in a controlled way after having already been affected by the early‑load issue on this hardware.

Bottom line: I'm happy to contribute real‑world data on the fix, just please bear with me if I'm a bit more careful than usual on a box that already hung at boot because of microcode.

[nero355]

There seems to be a connection to an outdated boot loader causing this issue when loading early.

Do we know how outdated it has to be in order to cause issues ?

[OPNenthu]

@OPNenthu

If you apply the steps outlined by me you do not need a boot loader update. That's the point. Similarly the changes by Franco that will be in 26.7.1 address exactly this issue for people without a current boot loader.

It would be interesting to see if a boot loader upgrade alone fixes things for you!

Ok, this is interesting.  Neither option works for me.

I first tried the bootcode update as per your post here (I have the same layout): https://forum.opnsense.org/index.php?topic=48145.msg243083#msg243083

On reboot, the console got stuck at the same place.

I then rolled back and applied your instructions in post #1, but this too got stuck.

Posting my install log for reference.  I wonder if there's a vendor/firmware quirk that breaks this?

Code Select Expand

root@firewall:~ # pkg install x86info
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following 3 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
libpci: 3.15.0 [OPNsense]
pciids: 20260624 [OPNsense]
x86info: 1.31.s03_1 [OPNsense]

Number of packages to be installed: 3

The process will require 2 MiB more space.
447 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/3] Fetching x86info-1.31.s03_1.pkg: 100%   72 KiB  73.4kB/s    00:01   
[2/3] Fetching pciids-20260624.pkg: 100%  297 KiB 304.2kB/s    00:01   
[3/3] Fetching libpci-3.15.0.pkg: 100%   79 KiB  80.5kB/s    00:01   
Checking integrity... done (0 conflicting)
[1/3] Installing pciids-20260624...
[1/3] Extracting pciids-20260624: 100%
[2/3] Installing libpci-3.15.0...
[2/3] Extracting libpci-3.15.0: 100%
[3/3] Installing x86info-1.31.s03_1...
[3/3] Extracting x86info-1.31.s03_1: 100%
=====
Message from x86info-1.31.s03_1:

--
===>   NOTICE:

This port is deprecated; you may wish to reconsider installing it:

Abandoned upstream, fails to identify anything remotely new according to upstream issue reports.

It is scheduled to be removed on or after 2025-06-30.
root@firewall:~ # pkg install cpu-microcode-intel
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
cpu-microcode-intel: 20260512_1 [OPNsense]

Number of packages to be installed: 1

The process will require 38 MiB more space.
25 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching cpu-microcode-intel-20260512_1.pkg: 100%   25 MiB  12.9MB/s    00:02   
Checking integrity... done (0 conflicting)
[1/1] Installing cpu-microcode-intel-20260512_1...
[1/1] Extracting cpu-microcode-intel-20260512_1: 100%
=====
Message from cpu-microcode-intel-20260512_1:

--
To enable microcode updates at boot time:
cat << EOF >> /boot/loader.conf
cpu_microcode_load="YES"
cpu_microcode_name="/boot/firmware/intel-ucode.bin"
EOF
root@firewall:~ # pkg install cpu-microcode-rc
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
cpu-microcode-rc: 1.0_2 [OPNsense]

Number of packages to be installed: 1

2 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching cpu-microcode-rc-1.0_2.pkg: 100%    2 KiB   2.5kB/s    00:01   
Checking integrity... done (0 conflicting)
[1/1] Installing cpu-microcode-rc-1.0_2...
[1/1] Extracting cpu-microcode-rc-1.0_2: 100%
=====
Message from cpu-microcode-rc-1.0_2:

--
This port includes an RC script, which is one of two methods to update
the CPU microcode on a FreeBSD system.

1. Early loading.
   This method does not use the RC script included here.
   This is the preferred method, because it ensures that any CPU features
   added or removed by a microcode update are visible to the kernel by
   applying the update before the kernel performs CPU feature detection.

   To enable updates using early loading, add the following lines to
   /boot/loader.conf:

   cpu_microcode_load="YES"

   and the appropriate one of these lines:

   cpu_microcode_name="/boot/firmware/intel-ucode.bin"
   cpu_microcode_name="/boot/firmware/amd-ucode.bin"

   The microcode update will be loaded when the system is rebooted.

   AMD systems running FreeBSD prior to 2024-02-22 snapshot
   34467bd76 only support late loading.


2. Late loading.
   This method, which does use the RC script included here, is enabled by
   adding the following line to /etc/rc.conf:

   microcode_update_enable="YES"

   The microcode update is then applied upon reboot or when the microcode
   update service is run via:

   # service microcode_update start

   If the CPU requires a microcode update, a console message such as the
   following will appear:

   Updating CPU Microcode...
   /usr/local/share/cpucontrol/m32306c3_00000022.fw: updating cpu /dev/cpuctl0 from rev 0x17 to rev 0x22... done.
   /usr/local/share/cpucontrol/m32306c3_00000022.fw: updating cpu /dev/cpuctl2 from rev 0x17 to rev 0x22... done.
   /usr/local/share/cpucontrol/m32306c3_00000022.fw: updating cpu /dev/cpuctl4 from rev 0x17 to rev 0x22... done.
   /usr/local/share/cpucontrol/m32306c3_00000022.fw: updating cpu /dev/cpuctl6 from rev 0x17 to rev 0x22... done.
   Done.

It is safe to enable both methods.
root@firewall:~ # pkg add -f https://pkg.opnsense.org/FreeBSD:15:amd64/snapshots/misc/os-cpu-microcode-intel-devel-1.1_1.pkg
Fetching os-cpu-microcode-intel-devel-1.1_1.pkg: 100%    1 KiB   1.3kB/s    00:01   
Installing os-cpu-microcode-intel-devel-1.1_1...
Extracting os-cpu-microcode-intel-devel-1.1_1: 100%
Reloading firmware configuration
Flushing all caches...done.
Writing firmware settings: OPNsense
Writing trust files...done.
Scanning /usr/share/certs/untrusted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
certctl: No changes to trust store were made.
Writing trust bundles...done.
Configuring login behaviour...done.
Configuring cron...done.
Configuring system logging...done.
root@firewall:~ #


[Patrick M. Hausen]

Do we know how outdated it has to be in order to cause issues?

Anything below the boot loader that comes with FreeBSD 15.1 is potentially suffering from this problem. It was fixed in this commit:

https://cgit.freebsd.org/src/commit/?h=releng/15.1&id=56e59980b673affb82c8c5da9d4338e10d946acd

If you do a fresh installation of OPNsense 26.7 you will have everything in place. If you came to 26.7 via an upgrade, you can update your boot loader from the running system by copying the /boot/loader.efi to your EFI partition.

Unfortunately or not OPNsense/Franco took the conservative approach and in the next OPNsense release we will simply load the ucode updates later in the boot process. I am not aware of any downsides to this approach with about 100 servers in production to back that claim. Yet, from a theoretical point of view loading ucode updates before kernel startup is "better". Not much arguing about that. We (me at least) are just not quite sure about the impact.

So there are two issues at hand, now:

- OPNsense as its own project will fix the boot lockup problem by changing to late ucode update loading.
- FreeBSD as upstream identified a problem with early ucode update loading and fixed it in the boot loader.

If I was product manager for OPNsense I would publish Franco's changes with the next release, case closed for now. But put a re-evaluation of the entire approach on the board for 27.1. Possibly (?) FreeBSD will have a reasonable unified approach to automated boot loader management.

Kind regards,
Patrick

[(MARLOO)]

@OPNenthu

Thanks for sharing your logs and screenshots, it's really helpful to see that on very similar hardware (N5105, 4× i226‑V) both the bootloader update and the "late load" steps still result in a boot hang.

Just to connect the dots: my original issue appeared after updating the i226‑V NVM firmware to 2.32 on 26.1.4. With the microcode plugin installed, the box would hang at boot when both WAN and LAN were linked. Removing the microcode plugin fixed it.

Out of curiosity, which NVM/firmware version are your i226‑V NICs currently running?
You can check with:

bash
dmesg | grep -i igc
and look for lines like "EEPROM Vx.xx‑x"

[OPNenthu]

@(MARLOO) I have no clue how or why the NVM version would impact this, but mine is on the original 2.13 that shipped with the mini pc.  Also, there may be differences in behavior across vendors and BIOS images, which was part of my question... so I am hesitant to assume that all N5105 boxes behave the same.  I'm not sure though.

[(MARLOO)]

@OPNenthu

Agreed, the NVM version itself probably doesn't directly "cause" the microcode issue. I mentioned it mainly because my boot hangs started right after I updated the i226 firmware, so I was trying to see if there was any pattern.

The reason I updated the i226‑V NVM to 2.32 back then was that the interfaces were flapping a lot. After the firmware update, the flapping completely disappeared. I documented that here:
https://forum.opnsense.org/index.php?topic=48695.195

The fact that you're on 2.13 and I was on 2.32, yet both of us hit boot hangs with the microcode plugin, actually supports your point: it's likely not about the NIC firmware, but about how early microcode loading interacts with the N5105 platform and/or specific BIOS images.

On my box, the practical workaround for now is to keep the microcode plugin removed and wait until the "late load" fix is officially rolled out. If you're okay with a bit more testing, your setup is probably perfect to help validate the fix across different N5105 boxes.

[db9]

So that's

https://github.com/opnsense/plugins/commit/ac68d4e
https://github.com/opnsense/tools/commit/854a5a8ce

for 26.7.1 then.

This seems to be the AMD version if I look at github, Intel will follow?

[Patrick M. Hausen]

So that's

https://github.com/opnsense/plugins/commit/ac68d4e
https://github.com/opnsense/tools/commit/854a5a8ce

for 26.7.1 then.

This seems to be the AMD version if I look at github, Intel will follow?

It's a major change for both versions and a minor fix for AMD.