I came across this mailing list thread while searching online about FreeBSD instabilities with N100, as many have been reporting upgrade issues. I'm not sure if this is related to the problematic microcode updates.
https://lists.freebsd.org/archives/freebsd-current/2025-January/006984.html
ChatGPT (for what it's worth) describes the issues like this:
Quote2. PCID / Cache Corruption Bug
The N100 has a known CPU erratum: INVLPG instruction with PCID enabled fails to flush TLB entries, causing data corruption on UFS file systems (sometimes panics or inode mangling) [ref] (https://lists.freebsd.org/archives/freebsd-current/2025-January/006984.html)
The workaround: add
vm.pmap.pcid_enabled=0
to loader.conf, ideally tested in production. Users report stability regained after disabling PCID [ref] (https://forums.freebsd.org/threads/unable-to-start-x-on-12th-gen-alder-lake-n100-with-failed-to-load-module-i915kms-while-kldstat-shows-loaded.92162/)
3. UFS Filesystem Instability
Severe issues such as inode corruption, filesystem panics, or UFS failure have been seen repeatedly when PCID remains enabled and UFS is used [ref] (https://lists.freebsd.org/archives/freebsd-current/2025-January/006984.html)
ZFS appears to avoid these issues entirely.
Quote⚠️ Why Might You Want to Disable It?
Some CPUs (including Intel N100/Alder Lake-N) exhibit hardware bugs when PCID is used. Specifically:
A known CPU erratum causes INVLPG (used to invalidate specific TLB entries) to fail when PCID is active.
This can result in stale or corrupted memory mappings, leading to:
Filesystem corruption (especially UFS)
Kernel panics
Data loss
Subtle stability problems
Disabling PCID (vm.pmap.pcid_enabled=0) avoids using the broken logic path.
🧪 Who Should Set It?
If you're using:
Intel N100 or other Alder Lake-N CPUs
UFS as a filesystem
FreeBSD 13.x or 14.x
👉 You should absolutely set vm.pmap.pcid_enabled=0 to ensure stability.
Seemed a little concerning and I thought I'd bring it up here for more technical insight.
I'm not affected personally as I don't have an N100 at this time.
ive got a n100 running on my workshop since aprox. 1 1/2 years without any issue with UFS and a local 128gb nvme in it. so i cant commit this issue.
I had similar issues with installing 25.7 (data corruption) installing the OS_CPU_MICROCODE_INTEL Plugin resolved the issue, previously was Not installed when I experienced the errors. Hooked up an monitor and noticed cluster errors during boot from upgrade (boot loop) reverted back to previous version installed the microcode plugin and then upgraded it has been running for an week now and several reboots no issues.
Maybe just use ZFS ?
Since the issue is sort of elusive on the CPU level chances are this affects stability in other ways than ZFS in particular (or any FS generally) so I think the recommendation for the tunable is something to consider for all relevant installs:
vm.pmap.pcid_enabled=0
I've also come to believe that moving way from our previous defaults hw.ibrs_disable=0 and vm.pmap.pti=1 back to FreeBSD's defaults (1 and 0 respectively) may cause some of the currently seen instabilities. Feel free to double check by setting these again on 25.7 and up:
hw.ibrs_disable=0
vm.pmap.pti=1
A number of people complained about OPNsense being slower than FreeBSD which was because of these security settings. From the looks of it now it has traded stability for speed on the lower end Intel CPUs for the most part.
https://docs.opnsense.org/troubleshooting/hardening.html#spectre-and-meltdown
Cheers,
Franco
Quote from: franco on August 05, 2025, 07:36:12 AMhw.ibrs_disable=0
vm.pmap.pti=1
I have an observation, though unrelated to the main topic.
These two tunables you mention appear in the OPNsense UI and are co-located in the default sorting (convenient for screenshotting) but they don't appear to have any values or defaults:
tunables.png
Corresponding 'sysctl':
root@firewall:~ # sysctl -a | grep -E 'vm.pmap.pcid_enabled|vm.pmap.pti|hw.ibrs_disable'
vm.pmap.pti: 0
vm.pmap.pcid_enabled: 0
hw.ibrs_disable: 1
I trust the sysctl output, just wondering why the OPNsense tunables list is that way? I get that the tunables list isn't complete and the system may support additonal ones not listed in OPNsense, but if they're listed they should have values I think (?).
----
Regarding Meltdown/Spectre, looks like a big can of worms trying to determine which particular CPU is or isn't vulnerable as some models vary depending even on the particular stepping. Uff.
Since the important one for data corruption is already disabled as recommended, I'm going to leave these alone. I'm not seeing any issues with performance or stability right now and I've kept up with UEFI and microcode updates. Probably these OS mitigations are more critical on virtualized environments where other things can be running, but I could be wrong (and wish to be corrected, as always).
Quote from: OPNenthu on August 05, 2025, 08:57:25 PMroot@firewall:~ # sysctl -a | grep -E 'vm.pmap.pcid_enabled|vm.pmap.pti|hw.ibrs_disable'
vm.pmap.pti: 0
vm.pmap.pcid_enabled: 0
hw.ibrs_disable: 1
I trust the sysctl output, just wondering why the OPNsense tunables list is that way? I get that the tunables list isn't complete and the system may support additonal ones not listed in OPNsense, but if they're listed they should have values I think (?).
We removed the explicit tunables from the config.xml, but we also removed the default values for it in order to go back to FreeBSD defaults. If you have these config.xml tunables but not set to "0" or "1" (meaning currently empty string "") they will use the system default now since we don't provide another default. Looking at your data that is the expected output on 25.7 when nothing else was specified.
Cheers,
Franco
Clear now, thanks!
Quote from: BrandyWine on August 05, 2025, 07:24:47 AMMaybe just use ZFS ?
Took a chance and backed up my config, fresh installed using ZFS and restored. Running fine thank you for heading me in the direction easier than I thought.
Hi Franco,
Quote from: franco on August 05, 2025, 07:36:12 AMSince the issue is sort of elusive on the CPU level chances are this affects stability in other ways than ZFS in particular (or any FS generally) so I think the recommendation for the tunable is something to consider for all relevant installs:
vm.pmap.pcid_enabled=0
I've also come to believe that moving way from our previous defaults hw.ibrs_disable=0 and vm.pmap.pti=1 back to FreeBSD's defaults (1 and 0 respectively) may cause some of the currently seen instabilities. Feel free to double check by setting these again on 25.7 and up:
hw.ibrs_disable=0
vm.pmap.pti=1
Sorry that I've captured this thread, don't know how to delete this post...
I've added vm.pmap.pcid_enabled=0 and corrected vm.pmap.pti=1 (was 0) but still it won't upgrade to 25.7.1.1_1
I also tried to install the intel-microcode-plugin (which I hadn't installed yet) but it claims that it'd need upgrade to 25.7.1_1 first, which doesn't work...
Trying to upgrade fails with:
Checking integrity...Assertion failed: (strcmp(uid, p->uid) != 0), function pkg_conflicts_check_local_path, file pkg_jobs_conflicts.c, line 315.
Child process pid=62294 terminated abnormally: Abort trap
Starting web GUI...done.
***DONE***
Any more iedaes?
Thanks and best regards,
Jochen
Drop to the console and do
# pkg install os-cpu-microcode-intel
and reboot to activate...
# opnsense-shell reboot
Then try the update again.
Cheers,
Franco
Installed microcode plugin parallel in both ways /boot/loader.conf and /etc/rc.conf but still no upgrade possible. UI said the microcode plugin was misconfigured so i removed it.
opnsense-bootstrap would be the last resort before a clean reinstall, but not knowing what is wrong there's not much more guidance to give and things would just continue to deteriorate for unknown reasons (like hardware failures).
Cheers,
Franco
FWIW, I migrated to a new N200 6-intell 226 made in China box for 25.7. Using default migration, I was having multiple random shutdowns per day. Per Franco, I reverted the tunables above to:
sysctl -a | grep -E 'vm.pmap.pcid_enabled|vm.pmap.pti|hw.ibrs_disable'
vm.pmap.pti: 1
vm.pmap.pcid_enabled: 0
hw.ibrs_disable: 0
and the box has been up and running without problems for 4 days now.
Don't want to take a victory lap just yet. Will post back once I hit the 1 week mark. Also note that I use cron to reboot box every night.
D
Quote from: OPNenthu on August 04, 2025, 08:35:26 PMI came across this mailing list thread while searching online about FreeBSD instabilities with N100, as many have been reporting upgrade issues. I'm not sure if this is related to the problematic microcode updates.
https://lists.freebsd.org/archives/freebsd-current/2025-January/006984.html (https://lists.freebsd.org/archives/freebsd-current/2025-January/006984.html)
Thank you!
I have an N100 system. I recently upgraded to 25.7. The system crashed during the first boot after the upgrade. I saw that there were file system errors during the boot. After a re-install it appeared to be running OK. While reading the forum to try and solve some other problems I had during the upgrade I found this thread. I have now added vm.pmap.pcid_enabled=0 to the tunables. Even though it seems to be running fine I assume that there could still have been some file system corruption. Do you think I should re-install 25.7? If so, how would I do this so that the vm.pmap.pcid_enabled=0 setting is in place before the first boot? Sorry this may be simple but I'm not very good with Linux.
Quote from: lmester on August 12, 2025, 06:41:42 AMEven though it seems to be running fine I assume that there could still have been some file system corruption. Do you think I should re-install 25.7? If so, how would I do this so that the vm.pmap.pcid_enabled=0 setting is in place before the first boot?
There's a chance that your disk has gone bad, which is something I see often on the forums. Try to install the 'os-smart' plugin and run a S.M.A.R.T check to see about your disk health. That plugin provides a simple status widget that you can add to the Lobby screen as well. Probably not worth reinstalling if it's running well now and passing the Health audit (System->Firmware->Status->Run an audit->Health).
As for tunables during installation, you can set them temporarily from the boot menu:
https://forum.opnsense.org/index.php?topic=47494.msg239887#msg239887
You'll need console access- serial or VGA.
Quote from: OPNenthu on August 12, 2025, 12:25:58 PMThere's a chance that your disk has gone bad, which is something I see often on the forums. Try to install the 'os-smart' plugin and run a S.M.A.R.T check to see about your disk health. That plugin provides a simple status widget that you can add to the Lobby screen as well. Probably not worth reinstalling if it's running well now and passing the Health audit (System->Firmware->Status->Run an audit->Health).
As for tunables during installation, you can set them temporarily from the boot menu:
https://forum.opnsense.org/index.php?topic=47494.msg239887#msg239887
You'll need console access- serial or VGA.
I think the system is corrupted. I'm seeing errors in the health audit. Can't install os-smart. Getting errors when I try to install updates.
############### Health audit errors ##############
***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 25.7 (amd64) at Tue Aug 12 09:20:05 EDT 2025
>>> Root file system: /dev/gpt/rootfs
>>> Check installed kernel version
Version 25.7 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 25.7 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense (Priority: 11)
>>> Check installed plugins
os-nut 1.9
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: ....
nspr-4.37: checksum mismatch for /usr/local/bin/nspr-config
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_aix32.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_aix64.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_darwin.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_freebsd.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_hpux32.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_hpux64.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_linux.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_netbsd.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_nto.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_openbsd.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_qnx.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_riscos.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_solaris.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/md/_win95.cfg
nspr-4.37: checksum mismatch for /usr/local/include/nspr/pratom.h
nspr-4.37: checksum mismatch for /usr/local/include/nspr/prinit.h
nspr-4.37: checksum mismatch for /usr/local/lib/libnspr4.a
nspr-4.37: checksum mismatch for /usr/local/lib/libnspr4.so
nspr-4.37: checksum mismatch for /usr/local/lib/libplc4.so
nspr-4.37: checksum mismatch for /usr/local/lib/libplds4.so
nspr-4.37: checksum mismatch for /usr/local/libdata/pkgconfig/nspr.pc
nspr-4.37: missing file /usr/local/share/licenses/nspr-4.37/LICENSE
nspr-4.37: missing file /usr/local/share/licenses/nspr-4.37/MPL20
nspr-4.37: missing file /usr/local/share/licenses/nspr-4.37/catalog.mk
Checking all packages.....
py311-certifi-2025.7.14: missing file /usr/local/lib/python3.11/site-packages/certifi-2025.7.14.dist-info/LICENSE
py311-certifi-2025.7.14: missing file /usr/local/lib/python3.11/site-packages/certifi-2025.7.14.dist-info/METADATA
py311-certifi-2025.7.14: missing file /usr/local/lib/python3.11/site-packages/certifi-2025.7.14.dist-info/RECORD
py311-certifi-2025.7.14: missing file /usr/local/lib/python3.11/site-packages/certifi-2025.7.14.dist-info/WHEEL
py311-certifi-2025.7.14: missing file /usr/local/lib/python3.11/site-packages/certifi-2025.7.14.dist-info/top_level.txt
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/__init__.py
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/__main__.py
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/__pycache__/__init__.cpython-311.opt-1.pyc
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/__pycache__/__init__.cpython-311.pyc
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/__pycache__/__main__.cpython-311.opt-1.pyc
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/__pycache__/__main__.cpython-311.pyc
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/__pycache__/core.cpython-311.opt-1.pyc
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/__pycache__/core.cpython-311.pyc
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/cacert.pem
py311-certifi-2025.7.14: checksum mismatch for /usr/local/lib/python3.11/site-packages/certifi/core.py
py311-certifi-2025.7.14: missing file /usr/local/share/licenses/py311-certifi-2025.7.14/LICENSE
py311-certifi-2025.7.14: missing file /usr/local/share/licenses/py311-certifi-2025.7.14/MPL20
py311-certifi-2025.7.14: missing file /usr/local/share/licenses/py311-certifi-2025.7.14/catalog.mk
Checking all packages.....
py311-typing-extensions-4.14.1: checksum mismatch for /usr/local/lib/python3.11/site-packages/__pycache__/typing_extensions.cpython-311.opt-1.pyc
py311-typing-extensions-4.14.1: checksum mismatch for /usr/local/lib/python3.11/site-packages/__pycache__/typing_extensions.cpython-311.pyc
py311-typing-extensions-4.14.1: missing file /usr/local/lib/python3.11/site-packages/typing_extensions-4.14.1.dist-info/METADATA
py311-typing-extensions-4.14.1: missing file /usr/local/lib/python3.11/site-packages/typing_extensions-4.14.1.dist-info/RECORD
py311-typing-extensions-4.14.1: missing file /usr/local/lib/python3.11/site-packages/typing_extensions-4.14.1.dist-info/WHEEL
py311-typing-extensions-4.14.1: missing file /usr/local/lib/python3.11/site-packages/typing_extensions-4.14.1.dist-info/licenses/LICENSE
py311-typing-extensions-4.14.1: checksum mismatch for /usr/local/lib/python3.11/site-packages/typing_extensions.py
Checking all packages..... done
>>> Check for core packages consistency
Core package "opnsense" at 25.7 has 68 dependencies to check.
Checking packages: .......................
opnsense-25.7 version mismatch, expected 25.7.1_1
Checking packages: ...........................
py311-duckdb-1.3.1_1 version mismatch, expected 1.3.2
Checking packages: ..............
sudo-1.9.17p1 version mismatch, expected 1.9.17p2
Checking packages: ..
syslog-ng-4.8.2_3 version mismatch, expected 4.8.2_4
Checking packages: ... done
***DONE***
######### os-smart install fails #########
***GOT REQUEST TO INSTALL***
Currently running OPNsense 25.7 (amd64) at Tue Aug 12 09:24:10 EDT 2025
Installation out of date. The update to opnsense-25.7.1_1 is required.
***DONE***
######## Firmware update fails ###########
***GOT REQUEST TO UPDATE***
Currently running OPNsense 25.7 (amd64) at Tue Aug 12 00:54:33 EDT 2025
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
Checking for upgrades (11 candidates): .......... done
Processing candidates (11 candidates): .......... done
The following 11 package(s) will be affected (of 0 checked):
Installed packages to be UPGRADED:
boost-libs: 1.88.0_1 -> 1.88.0_2
curl: 8.14.1 -> 8.15.0
ivykis: 0.43.2 -> 0.43.2_1
jq: 1.8.0 -> 1.8.1
libucl: 0.9.2_1 -> 0.9.2_2
nss: 3.113.1_1 -> 3.114
opnsense: 25.7 -> 25.7.1_1
py311-duckdb: 1.3.1_1 -> 1.3.2
sudo: 1.9.17p1 -> 1.9.17p2
syslog-ng: 4.8.2_3 -> 4.8.2_4
webp: 1.5.0 -> 1.6.0
Number of packages to be upgraded: 11
36 MiB to be downloaded.
[1/11] Fetching boost-libs-1.88.0_2.pkg: .......... done
[2/11] Fetching nss-3.114.pkg: .......... done
[3/11] Fetching jq-1.8.1.pkg: .......... done
[4/11] Fetching syslog-ng-4.8.2_4.pkg: .......... done
[5/11] Fetching webp-1.6.0.pkg: .......... done
[6/11] Fetching ivykis-0.43.2_1.pkg: .......... done
[7/11] Fetching curl-8.15.0.pkg: .......... done
[8/11] Fetching libucl-0.9.2_2.pkg: .......... done
[9/11] Fetching opnsense-25.7.1_1.pkg: .......... done
[10/11] Fetching py311-duckdb-1.3.2.pkg: .......... done
[11/11] Fetching sudo-1.9.17p2.pkg: .......... done
Checking integrity...Assertion failed: (strcmp(uid, p->uid) != 0), function pkg_conflicts_check_local_path, file pkg_jobs_conflicts.c, line 315.
Child process pid=26045 terminated abnormally: Abort trap
Starting web GUI...done.
***DONE***
QuoteAs for tunables during installation, you can set them temporarily from the boot menu:
https://forum.opnsense.org/index.php?topic=47494.msg239887#msg239887
I assume that i'll need to do this for all future updates unless the developers change the default for this tunable. Is this correct? This will make future updates painful :-(
Adding a persistent value to System: Settings: Tunables is all you need. This principle has not changed and if someone already had persistent values the update would not have changed them either.
Cheers,
Franco
sysctl -a | grep -E 'vm.pmap.pcid_enabled|vm.pmap.pti|hw.ibrs_disable'
vm.pmap.pti: 1
vm.pmap.pcid_enabled: 0
hw.ibrs_disable: 0
pkg clean -a
I still encounter an error when trying to update. Is it possible to upgrade without doing a clean install and restore config backup?
```
Checking integrity...Assertion failed: (strcmp(uid, p->uid) != 0), function pkg_conflicts_check_local_path, file pkg_jobs_conflicts.c, line 315.
Child process pid=37800 terminated abnormally: Abort trap
Starting web GUI...done.
***DONE***
```
I got sent this thread and though I think I understand what I'm ready, what I don't understand is "what are the final recommended tunables settings?" Each post flip-flops between 0|1 and I'm not sure what I should be applying.
What is recommended when on N100 and version 25.7.1_1-amd64
vm.pmap.pti: ?
vm.pmap.pcid_enabled: ?
hw.ibrs_disable: ?
--thanks
This thread can use a clean-up so here's my understanding of things:
People with Intel chips have been reporting problems during upgrade to 25.7, though the exact cause is not known. There are some people who are helped by uninstalling the intel microcode; some not. One person reported success after changing the tunables recommended here; others not. There doesn't seem to be a smoking gun and it's not even clear if everyone's issues are the same. Some people probably just have bad hardware or a corrupt filesystem (could be for various reasons, not necessarily due to this bug) which may be contributing to the reports.
That said-
The intel microcode workaround is discussed in other threads on the board, but as for the tunables in this thread:
vm.pmap.pcid_enabled=0
This ^ is the general recommendation for everyone on N-series chips. It's been shown to cause corruptions when enabled (per the FreeBSD mailing list topic), so keep it off. It's been defaulted to off in Linux as well. It may or may not fix anything you are experiencing, but it will at least help prevent potentially severe issues down the line.
hw.ibrs_disable: 0
vm.pmap.pti: 1
These ^ two are suggestions from Franco (https://forum.opnsense.org/index.php?msg=244032) in case you are still having stability issues, but YMMV. I didn't set these personally because I'm not having any problems with either stability or performance (I just wanted to prevent corruptions) and if you're not seeing any issue either then maybe leave these as they are.
The one poster (https://forum.opnsense.org/index.php?msg=244614) so far who reported some success after changing tunables had modified all three, so it's not clear which one made the difference for them.
@OPNenthu thanks for that nice summary :-)
Thanks for all these tipps but none of them helped me.
So, my question is:
can I just do a configuration backup of my running 25.7 , install the latest 25.7.2 from scratch and after that just restore from my backup or do I have to first install all plugins, that I'm using and then restore from backup? Or first restore and the install the plugins? What's the process?
BTW: the latest downloadable version is 25.7 (not 25.7.2!?) Will I have the same problems after new installation?
Thanks and best regards,
cibomato
Quote from: cibomato on September 05, 2025, 12:17:24 PMcan I just do a configuration backup of my running 25.7 , install the latest 25.7.2 from scratch and after that just restore from my backup
Yes, that's the way and I've done it a couple times.
Caveat emptor: this assumes your config file in intact and not corrupted.
Also, there could be changes from the versions that you had exported the config from which makes the config no longer compatible with newer versions, but I don't know if or how often this happens IRL. I'm thinking that care is taken by developers to preserve compatibility but I don't really know.
In your case I would check all of the release notes from your currently installed version to the present, looking for any potential areas where something might break. You can also try a VM install to test your config import on the latest version.
Quoteor do I have to first install all plugins, that I'm using and then restore from backup? Or first restore and the install the plugins? What's the process?
Install -> restore config -> reinstall plugins
QuoteBTW: the latest downloadable version is 25.7 (not 25.7.2!?) Will I have the same problems after new installation?
I don't think we can say for certain because we don't know what caused the problem for you in the first place. :(
Yes backup and re-install but make sure you are using zfs when re-installing.
Hey guys,
thanks for the hints!
New installation from scratch, restore config, install plugins and after that upgrading to 25.7.2 worked like charm!
Thank you very much and best regards,
cibomato