[SOLVED] System unable to boot after upgrade from 25.1.7 to 25.1.8_1

Started by Kewin, June 16, 2025, 06:09:33 PM

Previous topic - Next topic
June 16, 2025, 06:09:33 PM Last Edit: June 18, 2025, 10:29:16 PM by Kewin Reason: SOLVED
[SOLVED]: Uninstalled os-cpu-microcode-intel and system was able to boot without issues..

Hardware: Topton Intel N100 with 4xi226-V nics.

After upgrade from 25.1.7 to 25.1.8_1 the system is unable to boot using default/kernel.

Managed to get it up by selecting kernel.old at bootup.

Console from (failed) boot:
Loading /boot/loader.conf.local
\
/
-
            ______  _____  _____
            /  __  |/ ___ |/ __  |
            | |  | | |__/ | |  | |___  ___ _ __  ___  ___
            | |  | |  ___/| |  | / __|/ _ \ '_ \/ __|/ _ \
            | |__| | |    | |  | \__ \  __/ | | \__ \  __/
            |_____/|_|    |_| /__|___/\___|_| |_|___/\___|

 +-----------------------------------------+
 |                                        |          :::::::::::.
 |  1. Boot Multi user [Enter]            |          :::::::::::::.
 |  2. Boot Single user                    |      ::::              ::.
 |  3. Escape to loader prompt            |      ::::              ::::
 |  4. Reboot                              |      ::::              ::::
 |  5. Cons: Serial                        |      ::::              ::::
 |                                        |      ::::              ::::
 |  Options:                              |      ::::              ::::
 |  6. Kernel: default/kernel (1 of 2)    |      `::              ::::
 |  7. Boot Options                        |        `::::::::::::::
 |                                        |          `::::::::::::
 |  Loader needs to be updated            |
 +-----------------------------------------+
  Autoboot in 0 seconds. [Space] to pause        25.1 ``Ultimate Unicorn''  -

Loading kernel...
/boot/kernel/kernel text=0x181e58 text=0xe0fd88 text=0x45e14c data=0x180+0xe80 data=0x1967c0+0x469840 0x8+0x1a1de0+0x8+0x1c6db2
Loading configured modules...
/boot/entropy size=0x1000
/etc/hostid size=0x25
/boot/kernel/carp.ko size 0xfb90 at 0x216a000
/boot/kernel/zfs.ko size 0x5da630 at 0x217a000
/boot/kernel/if_lagg.ko size 0x16608 at 0x2755000
loading required module 'if_infiniband'
/boot/kernel/if_infiniband.ko size 0x3558 at 0x276c000
/boot/kernel/pfsync.ko size 0x11a18 at 0x2770000
loading required module 'pf'
/boot/kernel/pf.ko size 0x90960 at 0x2782000
/boot/kernel/pflog.ko size 0x3c10 at 0x2813000
/boot/kernel/if_bridge.ko size 0xea40 at 0x2817000
loading required module 'bridgestp'
/boot/kernel/bridgestp.ko size 0x8918 at 0x2826000
/boot/kernel/if_enc.ko size 0x4be0 at 0x282f000
/boot/firmware/intel-ucode.bin size=0xe65400
/boot/kernel/if_gre.ko size 0xaa30 at 0x3699000
/boot/kernel/opensolaris.ko size 0x1e280 at 0x36a4000
staging 0x5be00000 (not copying) tramp 0x6a656000 PT4 0x6a64d000
Start @ 0xffffffff80382000 ...
EFI framebuffer information:
addr, size    0x0, 0x0
dimensions    0 x 0
stride        0
masks          0x00000000, 0x00000000, 0x00000000, 0x00000000

Here it halts indefinitely!

When powercycling and booting kernel.old:
Loading /boot/loader.conf.local
\
/
-

            ______  _____  _____
            /  __  |/ ___ |/ __  |
            | |  | | |__/ | |  | |___  ___ _ __  ___  ___
            | |  | |  ___/| |  | / __|/ _ \ '_ \/ __|/ _ \
            | |__| | |    | |  | \__ \  __/ | | \__ \  __/
            |_____/|_|    |_| /__|___/\___|_| |_|___/\___|

 +-----------------------------------------+
 |                                        |          :::::::::::.
 |  1. Boot Multi user [Enter]            |          :::::::::::::.
 |  2. Boot Single user                    |      ::::              ::.
 |  3. Escape to loader prompt            |      ::::              ::::
 |  4. Reboot                              |      ::::              ::::
 |  5. Cons: Serial                        |      ::::              ::::
 |                                        |      ::::              ::::
 |  Options:                              |      ::::              ::::
 |  6. Kernel: kernel.old (2 of 2)        |      `::              ::::
 |  7. Boot Options                        |        `::::::::::::::
 |                                        |          `::::::::::::
 |  Loader needs to be updated            |
 +-----------------------------------------+
                                                  25.1 ``Ultimate Unicorn''

Loading kernel...
/boot/kernel.old/kernel text=0x181e58 text=0xe0fd88 text=0x45e14c data=0x180+0xe80 data=0x1967c0+0x469840 0x8+0x1a1de0+0x8+0x1c6db2
Loading configured modules...
/boot/kernel.old/if_bridge.ko size 0xea40 at 0x2169000
loading required module 'bridgestp'
/boot/kernel.old/bridgestp.ko size 0x8918 at 0x2178000
/boot/firmware/intel-ucode.bin size=0xe65400
/boot/kernel.old/zfs.ko size 0x5da630 at 0x2fe6000
/boot/kernel.old/opensolaris.ko size 0x1e280 at 0x35c1000
/boot/kernel.old/pfsync.ko size 0x11a18 at 0x35e0000
loading required module 'pf'
/boot/kernel.old/pf.ko size 0x90960 at 0x35f2000
/boot/entropy size=0x1000
/boot/kernel.old/if_lagg.ko size 0x16608 at 0x3684000
loading required module 'if_infiniband'
/boot/kernel.old/if_infiniband.ko size 0x3558 at 0x369b000
/boot/kernel.old/pflog.ko size 0x3c10 at 0x369f000
/boot/kernel.old/if_gre.ko size 0xaa30 at 0x36a3000
/boot/kernel.old/carp.ko size 0xfb90 at 0x36ae000
/etc/hostid size=0x25
/boot/kernel.old/if_enc.ko size 0x4be0 at 0x36be000
staging 0x5bc00000 (not copying) tramp 0x6a656000 PT4 0x6a64d000
Start @ 0xffffffff80382000 ...
EFI framebuffer information:
addr, size    0x0, 0x0
dimensions    0 x 0
stride        0
masks          0x00000000, 0x00000000, 0x00000000, 0x00000000
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.2-RELEASE-p3 stable/25.1-n269825-cbaf3aa6b26b SMP amd64
FreeBSD clang version 18.1.6 (https://github.com/llvm/llvm-project.git llvmorg-18.1.6-0-g1118c2e05e67)
VT(vga): resolution 640x480
CPU microcode: updated from 0xe to 0x1d
CPU: Intel(R) N100 (806.40-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0xb06e0  Family=0x6  Model=0xbe  Stepping=0
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x239ca7eb<FSGSBASE,TSCADJ,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA>
  Structured Extended Features2=0x98c007bc<UMIP,PKU,OSPKE,WAITPKG,GFNI,VAES,VPCLMULQDQ,RDPID,MOVDIRI,MOVDIR64B>
  Structured Extended Features3=0xfc184410<FSRM,MD_CLEAR,IBT,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x15c0fd6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO,TAA_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16334360576 (15577 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
WARNING: L3 data cache covers more APIC IDs than a package (7 > 3)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 3 2 1
random: entropy device external interface
wlan: mac acl policy registered
kbd0 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x75cc5000-0x75cc501e
smbios0: Version: 3.5, BCD Revision: 3.5
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <ALASKA A M I >
acpi0: Power Button (fixed)
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 19200000 Hz quality 950
Event timer "HPET" frequency 19200000 Hz quality 550
Event timer "HPET1" frequency 19200000 Hz quality 440
Event timer "HPET2" frequency 19200000 Hz quality 440
Event timer "HPET3" frequency 19200000 Hz quality 440
Event timer "HPET4" frequency 19200000 Hz quality 440
atrtc1: <AT realtime clock> on acpi0
atrtc1: Warning: Couldn't map I/O.
atrtc1: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x3000-0x303f mem 0x6000000000-0x6000ffffff,0x4000000000-0x400fffffff at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <XHCI (generic) USB 3.0 controller> mem 0x6001110000-0x600111ffff at device 13.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
xhci1: <XHCI (generic) USB 3.0 controller> mem 0x6001100000-0x600110ffff at device 20.0 on pci0
xhci1: 32 bytes context size, 64-bit DMA
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <AHCI SATA controller> port 0x3090-0x3097,0x3080-0x3083,0x3060-0x307f mem 0x81100000-0x81101fff,0x81103000-0x811030ff,0x81102000-0x811027ff at device 23.0 on pci0
ahci0: AHCI v1.31 with 1 6Gbps ports, Port Multiplier not supported
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igc0: <Intel(R) Ethernet Controller I226-V> mem 0x80e00000-0x80efffff,0x80f00000-0x80f03fff at device 0.0 on pci1
igc0: EEPROM V2.13-0 eTrack 0x80000284
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 4 RX queues 4 TX queues
igc0: Using MSI-X interrupts with 5 vectors
igc0: Ethernet address: 00:d0:b4:02:36:58
igc0: netmap queues/slots: TX 4/1024, RX 4/1024
pcib2: <ACPI PCI-PCI bridge> at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
igc1: <Intel(R) Ethernet Controller I226-V> mem 0x80b00000-0x80bfffff,0x80c00000-0x80c03fff at device 0.0 on pci2
igc1: EEPROM V2.13-0 eTrack 0x80000284
igc1: Using 1024 TX descriptors and 1024 RX descriptors
igc1: Using 4 RX queues 4 TX queues
igc1: Using MSI-X interrupts with 5 vectors
igc1: Ethernet address: 00:d0:b4:02:36:59
igc1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib3: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
igc2: <Intel(R) Ethernet Controller I226-V> mem 0x80800000-0x808fffff,0x80900000-0x80903fff at device 0.0 on pci3
igc2: EEPROM V2.13-0 eTrack 0x80000284
igc2: Using 1024 TX descriptors and 1024 RX descriptors
igc2: Using 4 RX queues 4 TX queues
igc2: Using MSI-X interrupts with 5 vectors
igc2: Ethernet address: 00:d0:b4:02:36:5a
igc2: netmap queues/slots: TX 4/1024, RX 4/1024
pcib4: <ACPI PCI-PCI bridge> at device 28.6 on pci0
pci4: <ACPI PCI bus> on pcib4
igc3: <Intel(R) Ethernet Controller I226-V> mem 0x80500000-0x805fffff,0x80600000-0x80603fff at device 0.0 on pci4
igc3: EEPROM V2.13-0 eTrack 0x80000284
igc3: Using 1024 TX descriptors and 1024 RX descriptors
igc3: Using 4 RX queues 4 TX queues
igc3: Using MSI-X interrupts with 5 vectors
igc3: Ethernet address: 00:d0:b4:02:36:5b
igc3: netmap queues/slots: TX 4/1024, RX 4/1024
pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci0
pci5: <ACPI PCI bus> on pcib5
nvme0: <Generic NVMe Device> mem 0x81000000-0x81003fff at device 0.0 on pci5
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
hdac0: <Intel Alder Lake-N HDA Controller> mem 0x6001120000-0x6001123fff,0x6001000000-0x60010fffff at device 31.3 on pci0
pci0: <serial bus> at device 31.5 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
ns8250: UART FCR is broken
ns8250: UART FCR is broken
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ns8250: UART FCR is broken
uart0: console (115200,n,8,1)
ns8250: UART FCR is broken
ns8250: UART FCR is broken
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> on acpi0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
atrtc0: Can't map interrupt.
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
Timecounter "TSC" frequency 806401197 Hz quality 1000
Timecounters tick every 1.000 msec
ugen1.1: <Intel XHCI root HUB> at usbus1
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus1
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
uhub1 on usbus0
uhub1: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
nvme0: Allocated 64MB host memory buffer
nda0 at nvme0 bus 0 scbus1 target 0 lun 1
nda0: <BKKJ nvme 256G V1226A5 2024012100933>
nda0: Serial Number 2024012100933
nda0: nvme version 1.3
nda0: 244198MB (500118192 512 byte sectors)
Trying to mount root from zfs:zroot/ROOT/default []...
uhub1: 3 ports with 3 removable, self powered
uhub0: 16 ports with 16 removable, self powered
Mounting filesystems...
pid 31 (zpool) is attempting to use unsafe AIO requests - not logging anymore
no pools available to import
Setting hostuuid: 4d5a20da-56ac-49ee-8728-bd647e79bfb3.
Setting hostid: 0x2d2ed7b9.
Configuring vt: keymap.
>>> Invoking import script 'importer'
Configuring crash dump device: /dev/nvd0p3
swapon: adding /dev/nvd0p3 as swap device
.ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.40/mach/CORE
32-bit compatibility ldconfig path:
done.
>>> Invoking early script 'upgrade'
>>> Invoking early script 'configd'
Starting configd.
>>> Invoking early script 'templates'
Generating configuration: OK
>>> Invoking early script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'dhcpleases'
>>> Invoking backup script 'duid'
>>> Invoking backup script 'netflow'
>>> Invoking backup script 'rrd'
>>> Invoking early script 'carp'
CARP event system: OK
Launching the init system...done.
Initializing..........done.
igc0: link state changed to UP
igc1: link state changed to UP
Starting device manager...ichsmb0: <Intel Alder Lake SMBus controller> port 0xefa0-0xefbf mem 0x6001128000-0x60011280ff at device 31.4 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_wmi0: <ACPI-WMI mapping> on acpi0
acpi_wmi0: cannot find EC device
acpi_wmi0: Embedded MOF found
ACPI: \_SB.WFDE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
acpi_wmi1: <ACPI-WMI mapping> on acpi0
acpi_wmi1: cannot find EC device
acpi_wmi1: Embedded MOF found
ACPI: \_SB.WFTE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
done.
Configuring login behaviour...done.
Configuring loopback interface..lo0: link state changed to UP
.done.
Configuring kernel modules...done.
Setting up extended sysctls...done.
Setting timezone: Europe/Copenhagen
Writing firmware settings: FreeBSD OPNsense
Writing trust files...done.
Scanning /usr/share/certs/untrusted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
certctl: No changes to trust store were made.
Writing trust bundles...done.
Setting hostname: OPNsense.lan
Generating /etc/resolv.conf...done.
Generating /etc/hosts...done.
Configuring system logging...done.
Configuring firewall.load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_aqm dn_aqm PIE loaded
......done.
Configuring hardware interfaces...done.
Configuring loopback interface...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring WAN_Untagged interface...done.
Configuring NPT_A0 interface...done.
Configuring DMZ interface...done.
Configuring Game interface...done.
Configuring Guest interface...done.
Configuring IoT interface...done.
Configuring LAB interface...done.
Configuring LAN interface...done.
Configuring Surveillance interface...done.
Configuring WAN interface...done.
Generating /etc/resolv.conf...done.
Generating /etc/hosts...done.
Configuring firewall.......done.
Configuring OpenSSH...done.
Starting web GUI...done.
Setting up routes...done.
Starting DHCPv4 service...done.
Starting DHCPv6 service...done.
Starting router advertisement service...done.
Starting Unbound DNS...done.
Configuring firewall.......done.
Setting up gateway monitor...done.
Configuring IPsec VPN...done.
Syncing OpenVPN settings...done.
Configuring WireGuard VPN...done.
Starting NTP service...done.
Configuring RFC 2136 clients...done.
Starting Unbound DNS...done.
>>> Invoking start script 'newwanip'
Reconfiguring IPv4 on vlan0.1.102
Reconfiguring IPv6 on vlan0.1.102
>>> Invoking start script 'freebsd'
osudpbroadcastrelay is not running.
Starting osudpbroadcastrelay.
osudpbroadcastrelay is not running.
Starting osudpbroadcastrelay.
osudpbroadcastrelay is not running.
Starting osudpbroadcastrelay.
timeout waiting for Tailscale service to enter a Running state; check health with "tailscale status"
Starting monit.
Starting Monit 5.35.2 daemon with http interface at /var/run/monit.sock
Starting ddclient_opn.
Starting snmpd.
rmdir: /var/etc/acme-client/home/deploy: Not a directory
rmdir: /var/etc/acme-client/home/dnsapi: Not a directory
rmdir: /var/etc/acme-client/home/notify: Not a directory
Starting acme_http_challenge.
Starting caddy... done
Log: /var/log/caddy/caddy.log
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled
Firewall rules loaded.
>>> Invoking start script 'syslog'
>>> Invoking start script 'carp'
>>> Invoking start script 'cron'
Starting Cron: OK
>>> Invoking start script 'openvpn'
>>> Invoking start script 'sysctl'
Service `sysctl' has been restarted.
>>> Invoking start script 'beep'
Root file system: zroot/ROOT/default
Mon Jun 16 17:44:58 CEST 2025

*** OPNsense.lan: OPNsense 25.1.8_1 (amd64) ***
--redacted--

Tried reinstalling kernel under packages, in case it somehow got broken during upgrade.
Intel N100, 4* I226-V, 16 GB mem, 256 GB NVMe