26.7.3 boot loop on Hyper-V guest -- anyone else?

[iorx]

Upgraded a Hyper-V VM (gen 2, synthetic NIC) from 26.7.2 to 26.7.3 and hit a restart loop. No panic output captured -- reverted to snapshot before I could gather logs.

Running: Windows Server Hyper-V, OPNsense amd64, VLANs on synthetic adapters (hn0--hn5).

Has anyone else on Hyper-V seen this with 26.7.3?

Was able to get this, it just flashes by... and then restart. Reverted back a snapshot of .2 to get it running again.

[Monviech (Cedrik)]

Whats your processor architecture? Intel, AMD, which generation?

This could be related or not, cannot see the backtrace.
https://github.com/opnsense/src/issues/297

[iorx]

Host:
Windows Server 2025 Standard (Build 26100), Hyper-V Generation 2 guest.

Host CPU:
AMD64 Family 23 Model 113 Stepping 0 (Zen 2 / Ryzen 7 3700X, ~3573 MHz). BIOS AMI P4.10 (2021-05-18).

Guest config:
8 vCPU, 5 GB static RAM, 6 synthetic NICs (vmbus/hn), VLANs handled via Hyper-V Access mode per adapter. No SR-IOV assigned to guest.

Relevant tunables active at time of upgrade:

debug.kassert.warn_only = 1
net.inet.rss.enabled = 1
net.inet.rss.bits = 3 (8 RSS queues matching 8 vCPUs)
net.isr.maxthreads = -1

I'm going to check your link, looks interesting. Trying to get a chance to turn it of and change vCPU to one.

[iorx]

Some more info.

Tried 26.7.3_8, showed the same problem.

Updated another system which went well:
Intel i7-8559U, 3 vCPU, I219-LM, Windows Server 2025

Claudes take on the information provided on the troublesome upgrade:
Both are Hyper-V Gen 2, Windows Server 2025, synthetic NICs (vmbus/hn), static RAM.

Worth noting: the AMD guest ran 26.7.2 (FreeBSD 15.1-p2) stably for weeks with the same 8-vCPU and RSS configuration. The boot loop only appeared with 26.7.3 (p3), which suggests a regression introduced in p2 → p3 rather than a pre-existing Hyper-V SMP issue. This also makes the Issue #297 link less likely to be the root cause, since that was reported against RC1 (p1) and should have manifested earlier.

Next planned test: disable RSS (net.inet.rss.enabled=0) before retrying the upgrade, to isolate whether RSS or vCPU count is the deciding factor.

Can't test this right now, but can give it a shoot on Sunday. That and the vCPU=1, but that should have shown for 26.7 already as I understand it.

[franco]

Data point: there were no Hyper-V changes in 26.7.3.


Cheers,
Franco

[iorx]

Hi again!

Think we maybe have to file this under strange

I set only one vCPU on the VM and ran the upgrade. It booted and works. Changed it back to 8 vCPU. And booted as it should into 26.7.3_8...

Still got a snapshot of 26.7.2 if we should investigate this further. I'm willing to revert and see if the problem can be reproduced.
I tried two times from 26.7.2 without success.

But, I did something different (without thinking about it...). The successful upgrade when I had set it to 1 vCPU was from the console, option 12. The previous two attempts when vCPU was set to 8 was done from UI. That shouldn't make any difference how the upgrade process is done, or?
 

[Monviech (Cedrik)]

If your guest has 8 vCPUs, and you issue this command and the kernel panic occurs, it's the same issue as in the ticket:

Code Select Expand

mount -r -t nullfs /usr/local/lib/python3.13 /mnt
This command should kernel panic /instantly/ so have a snapshot.

if not, then probably not

[franco]

The crash happens in file systems unionfs and nullfs from what we have seen.  If you turn off Unbound it should boot always (using nullfs).  We also saw this with the installer media (using unionfs).  Perhaps it's also not a 100% guaranteed crash.


Cheers,
Franco

[iorx]

Thanks guys for followup.

You want me to test anything out here?

DNS setup:

Unbound: enabled, port 8053, active on LAN and WG interfaces, forwards to Quad9 via DoT.
AdGuard Home (os-adguardhome-maxit): port 53, uses Unbound on 127.0.0.1:8053 as upstream for internal zones.
Dnsmasq: port 53053, DHCP hostname resolution only.

So Unbound is active and running at boot, which fits Franco's description of the nullfs trigger.

Would it add any value if I run the test (mount -r -t nullfs /usr/local/lib/python3.13 /mnt) on the running 26.7.3_8 system? I still have the 26.7.2 snapshot available if a confirmed reproduction on a clean upgrade attempt would be more useful.

[Monviech (Cedrik)]

Run the command and if the thing happens you have the same issue, if not then most likely not. Ensure you have more than one vcpu configured.

You dont need to test it if you don't want to. It would just be interesting I guess.

But if unbound already runs the nullfs kernel module is already loaded anyway so I wouldnt expect anything to happen.

If you wanne be totally sure try the 26.7 installer in a new VM with 2vCPUs to see if it kernel crashes when it boots.

In my experience it boots around 10% of the time and crashes 90%, so it might be a race and not 100%.