26.7.3 boot loop on Hyper-V guest -- anyone else?

[iorx]

Upgraded a Hyper-V VM (gen 2, synthetic NIC) from 26.7.2 to 26.7.3 and hit a restart loop. No panic output captured -- reverted to snapshot before I could gather logs.

Running: Windows Server Hyper-V, OPNsense amd64, VLANs on synthetic adapters (hn0--hn5).

Has anyone else on Hyper-V seen this with 26.7.3?

Was able to get this, it just flashes by... and then restart. Reverted back a snapshot of .2 to get it running again.

[Monviech (Cedrik)]

Whats your processor architecture? Intel, AMD, which generation?

This could be related or not, cannot see the backtrace.
https://github.com/opnsense/src/issues/297

[iorx]

Host:
Windows Server 2025 Standard (Build 26100), Hyper-V Generation 2 guest.

Host CPU:
AMD64 Family 23 Model 113 Stepping 0 (Zen 2 / Ryzen 7 3700X, ~3573 MHz). BIOS AMI P4.10 (2021-05-18).

Guest config:
8 vCPU, 5 GB static RAM, 6 synthetic NICs (vmbus/hn), VLANs handled via Hyper-V Access mode per adapter. No SR-IOV assigned to guest.

Relevant tunables active at time of upgrade:

debug.kassert.warn_only = 1
net.inet.rss.enabled = 1
net.inet.rss.bits = 3 (8 RSS queues matching 8 vCPUs)
net.isr.maxthreads = -1

I'm going to check your link, looks interesting. Trying to get a chance to turn it of and change vCPU to one.

[iorx]

Some more info.

Tried 26.7.3_8, showed the same problem.

Updated another system which went well:
Intel i7-8559U, 3 vCPU, I219-LM, Windows Server 2025

Claudes take on the information provided on the troublesome upgrade:
Both are Hyper-V Gen 2, Windows Server 2025, synthetic NICs (vmbus/hn), static RAM.

Worth noting: the AMD guest ran 26.7.2 (FreeBSD 15.1-p2) stably for weeks with the same 8-vCPU and RSS configuration. The boot loop only appeared with 26.7.3 (p3), which suggests a regression introduced in p2 → p3 rather than a pre-existing Hyper-V SMP issue. This also makes the Issue #297 link less likely to be the root cause, since that was reported against RC1 (p1) and should have manifested earlier.

Next planned test: disable RSS (net.inet.rss.enabled=0) before retrying the upgrade, to isolate whether RSS or vCPU count is the deciding factor.

Can't test this right now, but can give it a shoot on Sunday. That and the vCPU=1, but that should have shown for 26.7 already as I understand it.

[franco]

Data point: there were no Hyper-V changes in 26.7.3.


Cheers,
Franco

[iorx]

Hi again!

Think we maybe have to file this under strange

I set only one vCPU on the VM and ran the upgrade. It booted and works. Changed it back to 8 vCPU. And booted as it should into 26.7.3_8...

Still got a snapshot of 26.7.2 if we should investigate this further. I'm willing to revert and see if the problem can be reproduced.
I tried two times from 26.7.2 without success.

But, I did something different (without thinking about it...). The successful upgrade when I had set it to 1 vCPU was from the console, option 12. The previous two attempts when vCPU was set to 8 was done from UI. That shouldn't make any difference how the upgrade process is done, or?
 

[Monviech (Cedrik)]

If your guest has 8 vCPUs, and you issue this command and the kernel panic occurs, it's the same issue as in the ticket:

Code Select Expand

mount -r -t nullfs /usr/local/lib/python3.13 /mnt
This command should kernel panic /instantly/ so have a snapshot.

if not, then probably not

[franco]

The crash happens in file systems unionfs and nullfs from what we have seen.  If you turn off Unbound it should boot always (using nullfs).  We also saw this with the installer media (using unionfs).  Perhaps it's also not a 100% guaranteed crash.


Cheers,
Franco

[iorx]

Thanks guys for followup.

You want me to test anything out here?

DNS setup:

Unbound: enabled, port 8053, active on LAN and WG interfaces, forwards to Quad9 via DoT.
AdGuard Home (os-adguardhome-maxit): port 53, uses Unbound on 127.0.0.1:8053 as upstream for internal zones.
Dnsmasq: port 53053, DHCP hostname resolution only.

So Unbound is active and running at boot, which fits Franco's description of the nullfs trigger.

Would it add any value if I run the test (mount -r -t nullfs /usr/local/lib/python3.13 /mnt) on the running 26.7.3_8 system? I still have the 26.7.2 snapshot available if a confirmed reproduction on a clean upgrade attempt would be more useful.

[Monviech (Cedrik)]

Run the command and if the thing happens you have the same issue, if not then most likely not. Ensure you have more than one vcpu configured.

You dont need to test it if you don't want to. It would just be interesting I guess.

But if unbound already runs the nullfs kernel module is already loaded anyway so I wouldnt expect anything to happen.

If you wanne be totally sure try the 26.7 installer in a new VM with 2vCPUs to see if it kernel crashes when it boots.

In my experience it boots around 10% of the time and crashes 90%, so it might be a race and not 100%.

[iorx]

Hi again!

Updated again, to _11, this update didn't require a reboot but did one anyway.
Ended up in a bootloop. Tried kern.panic_reboot_wait_time=-1 too, but that didn't work, no stop to see what actually happen.

So, a MacGyver solution was to record the console 🙂

Got this (extracted from video frames):

Code Select Expand

panic: page fault
cpuid = 3

hv_hid_probe() at hv_hid_probe/frame 0xfffffe0070bd8a90
device_probe() at device_probe+0x71
bus_generic_driver_added() at bus_generic_driver_added+0x67
devclass_driver_added() at devclass_driver_added+0x29
device_do_deferred_actions() at device_do_deferred_actions+0x3b
devctl2_ioctl() at devctl2_ioctl+0x211
devfs_ioctl() at devfs_ioctl+0xcb
vn_ioctl() at vn_ioctl+0xc8
kern_ioctl() at kern_ioctl+0x286
sys_ioctl() at sys_ioctl+0x101
amd64_syscall() at amd64_syscall+0x126
fast_syscall_common() at fast_syscall_common+0xf8
To add:

• Looks like it always manage to boot OK when vCPU=1
• vCPU=8 sometimes manage to boot but it can take some time before it succeeds...

[iorx]

LLM generated:

Compared loader.conf between the two systems. Key differences:

Code Select Expand

                              Boot loop system    Test VM (OK)
net.inet.rss.enabled          1                   not set (=0)
net.inet.rss.bits             3                   not set
net.isr.dispatch              deferred            not set
net.isr.maxthreads            -1                  not set

RSS is the most notable difference. The test VM has none of those set, the production system has RSS enabled with 8 queues and net.isr.maxthreads=-1, which increases the number of active threads at boot and likely widens the race window for the hv_hid_probe page fault.
Next test: disable RSS (net.inet.rss.enabled=0, remove net.inet.rss.bits and net.isr.maxthreads) on the production system and retry with 8 vCPU.

Can't test right now, family is online 😁

But have rebooted the newly installed 27.7.3_11 a couple of times and it also boot looped now. Interesting.
Behavior, it loops a couple of times and then manage to do a complete boot.