How to enable automatic microcode updates

[meyergru]

As many firewall appliances suffer from neglect by they manufacturers, they often lack current microcode updates.

For example, Intel Alder Lake based CPUs (like N100, N200, N300 and I1215U have a bug that cause instabilities.

As of OpnSense 24.7.2, microcode updates can be enabled for AMD and Intel CPUs by installing the corresponding plugin os-cpu-microcode-amd or os-cpu-microcode-intel (not both!). The update will be done automatically upon next reboot.

The old post is left here for reference:


Since 23.7.9, there should be microcode updates available, which, when enabled, should cure those problems.

Though OpnSense does not have those enabled per default (e.g. it obviously is not applicable for VM installations), the neccessary tools are provided. Here is how:

1. Install the package cpu-microcode - this is a newer variant of the old devcpu-data package. From a shell, call:

Code: [Select]

echo y | pkg install cpu-microcode

Alongside with this package, the RC scripts and firmwares for both Intel and AMD CPUs will be installed. If you had the devcpu-data package installed before, it will get replaced.

2. Use the web UI to create these two tuneables in /boot/loader.conf:

Code: [Select]

cpu_microcode_load="YES"
cpu_microcode_name="/boot/firmware/intel-ucode.bin"

Please double-check that the entires exist. Note that this is only for Intel CPUs, because this addresses only the early boot stage, which is not available for AMD CPUs. This can be fixed in the next step.

3. Add an entry to /etc/rc.conf:

Code: [Select]

echo 'microcode_update_enable="YES"' >> /etc/rc.conf

This will enable a RC script which handles both Intel and AMD CPUs during boot.

4. (Optional and only available from 23.7.6 on)

Code: [Select]

pkg install x86info
rehash
x86info -a | fgrep -i microcode

The last command will show you the current microcode version.

5. Reboot or call the CPU firmware update script once with:

Code: [Select]

service microcode_update start

You will see an error if you skipped step 3.

6. (Optional and only available from 23.7.6 on) Call:

Code: [Select]

kldload -q cpuctl; x86info -a | fgrep -i microcode
again and you will probably see a different version than in step 4.

Note that these settings and installation of packages will not be part of a configuration backup, so after a hardware migration, you have to repeat most of it.

These instructions are provided as-is, so no warranties whatsoever.

[franco]

It might be possible to wrap this into a trivial plugin, but the recent packaging change in FreeBSD made me happy that we didn't sink time here before. For now I just switched to the new packages. The old ones are no longer available.


Cheers,
Franco

[newsense]

Thanks meyergru, was just about to look into it (couldn't remember if I installed the meta anywhere) - when I saw Franco's answer which sent me searching for answers.

So...speaking of packages...

Code: [Select]

root@OPNsense:~ # pkg search cpu-microcode

cpu-microcode-1.0              Meta-package for CPU microcode updates
cpu-microcode-amd-20230808     AMD CPU microcode updates
cpu-microcode-intel-20230808   Intel CPU microcode updates
cpu-microcode-rc-1.0           RC script for CPU microcode updates


It loos like the RC package would be pulled along with the rest

Code: [Select]

root@OPNsense:~ # pkg install cpu-microcode
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating mimugmail repository catalogue...
mimugmail repository is up to date.
All repositories are up to date.
The following 4 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        cpu-microcode: 1.0 [OPNsense]
        cpu-microcode-amd: 20230808 [OPNsense]
        cpu-microcode-intel: 20230808 [OPNsense]
        cpu-microcode-rc: 1.0 [OPNsense]

Number of packages to be installed: 4


So we'd get this script installed in /usr/local/etc/rc.d/microcode_upddate

#!/bin/sh

# PROVIDE:   microcode_update
# REQUIRE:   root mountcritlocal
# KEYWORD:   nojail
# BEFORE:   SERVERS

#
# Add the following line to /etc/rc.conf to enable flow-capture:
# microcode_update_enable (bool):   Set it to "YES" to update microcode on startup
#               Set to "NO" by default.
# microcode_update_datadir (str):   Directory, microcode updates stored in.
#               Default is "/usr/local/share/cpucontrol"
# microcode_update_cpus (str):      A list of cpus to update on startup, or "ALL" for all.
#               Example: microcode_update_cpus="0 1"
#               Set to "ALL" by default.
# microcode_update_flags (str):      Flags for cpucontrol(.

. /etc/rc.subr

name="microcode_update"
rcvar=microcode_update_enable
stop_cmd=":"
start_cmd="microcode_update_start"
required_modules="cpuctl"

CMT="/usr/sbin/cpucontrol"

microcode_update_start()
{
   echo "Updating CPU Microcode..."
   if [ "${microcode_update_cpus}" = "ALL" ]; then
      ncpu=`/sbin/sysctl -n hw.ncpu`
      cpus=`jot ${ncpu} 0`;
   else
      cpus=${microcode_update_cpus}
   fi
   for i in ${cpus}; do
      ${CMT} -u ${microcode_update_flags} \
                    -d "${microcode_update_datadir}" /dev/cpuctl${i} 2>&1 | \
                    logger -p daemon.notice -t microcode_update || \
          (echo "Microcode Update Failed." && exit 1)
   done
   if [ "${microcode_update_cpus}" = "ALL" ]; then
                CPUCONTROL_UPDATED=$(cpucontrol -h 2>&1 | grep -q -- -e; echo $?)
                if [ ${CPUCONTROL_UPDATED} -ne 0 ]; then
                        echo "Please update your system in order to update CPU microcode."
                else
         ${CMT} -e /dev/cpuctl0 >/dev/null 2>&1
         if [ $? -ne 0 ]; then
            echo "Re-evalulation of CPU flags Failed."
            exit 1
         fi
                fi
   fi
   echo "Done."
}

load_rc_config $name

# Set default values
if [ -n "${microcode_cpus}" ]; then
   if [ -n "${microcode_update_cpus}" ]; then
      echo "Warning: Ignoring deprecated rc variable, microcode_cpus."
   else
      echo "Warning: rc variable microcode_cpus is deprecated.
Warning: Set microcode_udpate_cpus instead."
      microcode_update_cpus="${microcode_cpus}"
        fi
fi

: ${microcode_update_enable="NO"}
: ${microcode_update_datadir="/usr/local/share/cpucontrol"}
: ${microcode_update_cpus="ALL"}
: ${microcode_update_flags=""}

run_rc_command "$1"

If we're to stick as close as possible to FreeBSD maybe the best path forward would be to include x86info in OPNsense if there aren't a ton of unneeded dependencies or it ? Any future plugin would likely depend on it.


Thinking about this issue with a security hat --- plugin or no plugin, I would much rather have cpu-microcode in the default packages to be installed when deploying OPNsense on AMD64 - and have it pulled automatically in an upcoming 23.7.x

[franco]

I'm not seeing this as a core requirement. The fact that it isn't a plugin yet gives an insight into how important it is for the average user.

And happy to include x64info package given it's not on a heavy dependency chain.


Cheers,
Franco

[meyergru]

Yes, all needed packages are drawn in alongside cpu-microcode.

I am all for including x86info, as this eliminates the need to fetch a FreeBSD package - although it is only needed for verification (if an update was even done).

As for including cpu-microcode per default: It is not applicable for VM installations, so I second that it is not a core requirement.

[franco]

https://github.com/opnsense/tools/commit/eebe1f8fa

[newsense]

Thanks Franco, I just found 2.7.5 so I guess it was a timing issue and it will become available in 23.7.6

As for including cpu-microcode per default: It is not applicable for VM installations, so I second that it is not a core requirement.

Whether users at large understand or care about what cpu-microcode does is less important. Including the package in the deployment list would provide a secure default, otherwise is just an option.

Since none of the os-xen, os-vmware or os-virtualbox are defaults cpu-microcode can be marked for removal as soon as a user tries to install any of the aforementioned packages.

The only place where cpu-microcode would need to be excluded would likely be in the BE vm


I rest my case 

[moonman]

So this update pkg to 1.20.x and it now gives an error checking for new packages/update. Does anyone know how to rollback to the previous version of pkg. I have the package itself locally from a different system, just can't figure out how to properly downgrade.

4. (Optional) If you want to verify that the updates are working, you can install the package x86info. This is not contained in OpnSense, therefore you have to edit /usr/local/etc/pkg/repos/FreeBSD.conf to enable FreeBSD repositories temporarily. You can use these commands:

Code: [Select]

echo "FreeBSD: { enabled: yes }" > /usr/local/etc/pkg/repos/FreeBSD.conf
echo y | pkg install x86info"
echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
rehash
kldload -q cpuctl
x86info -a | fgrep -i microcode


This is the log

Code: [Select]

root@OPNsense:~ # echo y | pkg install x86info
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    7 MiB   6.9MB/s    00:01
Processing entries: 100%
FreeBSD repository update completed. 34062 packages processed.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.19.2 -> 1.20.7 [FreeBSD]

Number of packages to be upgraded: 1

The process will require 21 MiB more space.
9 MiB to be downloaded.

[1/1] Fetching pkg-1.20.7.pkg: 100%    9 MiB   9.0MB/s    00:01    %
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.19.2 to 1.20.7...
[1/1] Extracting pkg-1.20.7: 100%
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating OPNsense repository catalogue...
pkg: No SRV record found for the repo 'OPNsense'
pkg: packagesite URL error for pkg+http://mirror.sfo12.us.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg -- pkg+:// implies SRV mirror type
pkg: packagesite URL error for pkg+http://mirror.sfo12.us.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.txz -- pkg+:// implies SRV mirror type
Unable to update repository OPNsense
Error updating repositories!
root@OPNsense:~ # echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
root@OPNsense:~ # rehash
root@OPNsense:~ # kldload -q cpuctl
root@OPNsense:~ # pkg update
Updating OPNsense repository catalogue...
pkg: No SRV record found for the repo 'OPNsense'
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
pkg: packagesite URL error for pkg+http://mirror.sfo12.us.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.pkg -- pkg+:// implies SRV mirror type
pkg: packagesite URL error for pkg+http://mirror.sfo12.us.leaseweb.net/opnsense/FreeBSD:13:amd64/23.7/latest/packagesite.txz -- pkg+:// implies SRV mirror type
Unable to update repository OPNsense
Error updating repositories!
@meyergru I would delete that part before someone else stumbles upon it.

[meyergru]

@moonman: Whatever is your problem has nothing to do with the installation of x86info package itself, but obviously your pkg has been updated. You can check with: "pkv -v". I have 1.19.2, I do not know why yours was updated at all (mine did not and still does not).

The error message clearly is for the OPNsense repository. That means your configuration is incompatible with the new pkg version.

BTW: I just saw this: https://forum.opnsense.org/index.php?topic=36333.0


First, you should change the repository parameters under /usr/local/etc/pkg/repos/OPNsense.conf in order to be able to use the OPNsense repository again. See: https://www.reddit.com/r/freebsd/comments/15bghf5/pkg_no_srv_record_found_for_the_repo_freebsd/

That is, either remove the "pkg+" part of the repo URL if it is present or change the mirror_type from "SRV" to "NONE". I guess this will be fixed centrally already from what Franco wrote.

Then, you should be able to use "pkg update" again and probably downgrade somehow. Usually, the old package should still be in /var/cache/pkg.

Remember to change back the repo URL or mirror type afterwards or wait for the SRV records being changed as Franco announced.


I still changed the installation step as x86info will be present in 23.7.6 anyhow and get rid of the FreeBSD dependency.

[franco]

Yeah, pkg 1.20 broke a working situation by enforcing a stricter setup. Quarterly updates hit beginning of October. Same as usual. Expect a lot more of the same as people run into it.

We will be staying on 1.19 for a while longer for this and other reasons.


Cheers,
Franco

[moonman]

Thanks everyone. Running

Code: [Select]

pkg add -f /var/cache/pkg/pkg-1.19.2.pkgfixed it for me

[Ground_0]

Not quite a necro, I hope..

1. How vulnerable to microcode attacks are those of us using a bare metal install of OPNsense with no remote login?
2. Should those of us on bare metal with no remote login consider enabling automatic microcode updates?

My (limited) understanding is that side channel attacks like Spectre and Meltdown are a threat if you run applications on a system designed for multiple untrusted users, like in a webserver environment. I also fear that configuring automatic updates (IF they are not strictly necessary on an OPNsense machine without remote login) might lead to "entropy" wherein one might be met with unexpected breakage with updates and forget how to address it. It seems that the packages and configuration for this have changed over time and may continue to change and require manual intervention.

Any advice appreciated.

[meyergru]

If you limit the scope to these specific CPU bugs, you could argue that nobody could sniff into running processes anyway, given that they can start no program on the firewall. However, some microcode updates address stability, performance and / or real CPU bugs. Here is an example.

They were invented after the infamous Intel Pentium FDIV and F00F bugs.

Some of these updates are not even discussed openly.

AMD is known to often address issues with newer AGESA versions. Intel is as well. Alas, both companies sometimes miss to publish new fixes in time, thus official Linux and FreeBSD microcode repositories lag behind. There is a repository on Github that contains all known microcodes, but they are not in a form to use them directly.

For these reasons, I apply such upgrades even if there are no specific security issues, just for good measure. Sometimes I even patch the firmwares to include fixes for known defects myself, but that is beyond the scope of what normally needs to be done.

[Seimus]

Thank you for the guide,

I did this for N5105 as well N100 worked flawlessly.

Regards,
S.

[hushcoden]

I've just installed that according to the guide, and if I understood correctly, it can be done in three steps:

 1. From a shell: echo y | pkg install cpu-microcode

 2. Use the web UI to create these two tuneables in /boot/loader.conf:

     cpu_microcode_load="YES"

     cpu_microcode_name="/boot/firmware/intel-ucode.bin"   -> for Intel CPUs

     or

     cpu_microcode_name="/boot/firmware/amd-ucode.bin"   -> for AMD CPUs

3. Reboot

Is that right?

Tia.