[SOLVED] Performance issue after upgrading to 26.7 on a KVM-backed VM

[lesterd88]

Hi all,

Sharing this here as I originally posted on the subreddit, and was pointed to the forum to share in case anyone else has the same issue. Full disclosure, Claude helped me track this down and I asked it to write this summary up of the situation:

To note, I'm using HPE VM Essentials, which is a KVM based hypervisor.

# OPNsense 26.7 (FreeBSD 15.1) on KVM: LAPIC eventtimer miscalibration causes ~65k interrupts/sec/core — full diagnosis + mitigation

**TL;DR:** After upgrading a KVM guest from 26.1 to 26.7, the VM became nearly unusable (GUI crawling, dashboard widgets failing, configd repeatedly dying on start). Root cause: FreeBSD 15.1 no longer picks up the Hyper-V paravirtual timecounters that KVM's Hyper-V enlightenments provide (14.x on an identical VM does), and the LAPIC eventtimer gets calibrated at **416 kHz** — roughly 1000x low — so one-shot mode storms at **~65,000 timer interrupts/sec/core**. Every interrupt is a vmexit; the CPU tax made everything downstream fail. Mitigation: `kern.eventtimer.timer=i8254`. Fully reproducible, all numbers below.

## Environment

- OPNsense 26.7 (fresh upgrade from 26.1, day-one GA)
- QEMU/KVM guest (HPE VM Essentials / libvirt-based), Hyper-V enlightenments enabled (stack default)
- 4 vCPU, host CPU Xeon E5-2650v2 (Ivy Bridge)
- Backup node of a CARP pair — its peer is an **identical VM on the same host still on 26.1**, which made a clean A/B comparison possible
- Integrity verified clean before blaming the kernel: `pkg check -sa` zero mismatches, `zpool status` zero errors

## Symptoms after upgrade

- GUI extremely sluggish, dashboard widgets timing out / showing "failed"
- `configctl system status` taking 8+ seconds or hanging
- configd repeatedly killed during startup (rc wrapper / socket-wait timeouts giving up before it finished initializing)
- Sustained high ambient CPU with no traffic (node is CARP BACKUP, passing nothing)

## Diagnosis

### 1. Timer interrupt storm

```
# 26.7 node (FreeBSD 15.1):
vmstat -i | grep timer
cpu0:timer   1721502843   65470
cpu1:timer   1648644578   62699
cpu2:timer   1658830161   63087
cpu3:timer   1642275851   62457
```

~65k/sec **per core**, ~260k/sec for the VM, suspiciously pinned near 2^16. The identical VM on 26.1 (FreeBSD 14.x):

```
cpu0:timer   433639790   178
cpu1:timer   240119943    99
cpu2:timer   225831754    93
cpu3:timer   221488530    91
```

### 2. Hyper-V timecounters missing on 15.1

```
# 26.1 / FreeBSD 14.x:
kern.timecounter.hardware: Hyper-V-TSC
kern.timecounter.choice: TSC-low(-100) i8254(0) ACPI-fast(900) HPET(950) Hyper-V-TSC(3000) Hyper-V(2000) dummy(-1000000)

# 26.7 / FreeBSD 15.1, same host, same VM shape, same enlightenments:
kern.timecounter.hardware: HPET
kern.timecounter.choice: TSC-low(-100) i8254(0) ACPI-fast(900) HPET(950) dummy(-1000000)
```

The Hyper-V entries aren't just losing the priority contest — they're **absent from the choice list**, i.e., the paravirtual clock support never attached on 15.1.

### 3. LAPIC eventtimer calibrated ~1000x low

```
kern.eventtimer.et.LAPIC.frequency: 416000        # 416 kHz (!)
machdep.tsc_freq: 2599903050                      # TSC calibrated correctly at 2.6 GHz
```

KVM's emulated APIC timer runs at the ~1 GHz bus clock; the kernel believes 416 kHz. Every one-shot interval it programs is ~1000x too short → interrupt storm. TSC calibration is fine — only the LAPIC calibration is broken, consistent with a codepath that lost its paravirtual reference clock.

Supporting data point: forcing `kern.eventtimer.periodic=1` at hz=100 produced ~5,000 ints/sec/core instead of the expected ~100 — again a large constant factor too fast, consistent with the same miscalibration rather than load.

### 4. Cascade into userland

The storm's vmexit tax inflated cold Python 3.13 imports of configd's module set to ~7–8s (measured via truss + timing), exceeding the start-race patience of the rc wrapper / configctl socket-wait / dashboard polls, which then killed the half-started daemon — producing the "configd won't start" symptom. (A secondary issue on this box: the upgrade left the bytecode cache unpopulated, worsening cold imports; `python3 -m compileall` fixed that part. Note that compileall "errors" under `/usr/local/lib/python3.13/test/` are intentional bad-syntax test fixtures, not corruption.)

## Mitigation (validated)

System → Settings → Tunables: **`kern.eventtimer.timer = i8254`**

The i8254 PIT has an architecturally fixed frequency, so it can't be miscalibrated. Post-change and post-reboot:

```
irq0: attimer0   34196   395      # ~400/sec total for the whole VM (4×hz via IPI forwarding)
cpu0:timer — zero interrupts since boot
time configctl system status → 0.18s   (was 8.4s)
configd running supervised, PID 596, wins its start race cold
```

Interrupt load reduced ~650x; the box is completely normal again.

(HPET is not an option here — QEMU's HPET exposes no FSB/MSI interrupt path FreeBSD accepts, so it registers only as a timecounter, not an eventtimer. Eventtimer choices on this guest are LAPIC(600) i8254(100) RTC(0).)

## Reproduction

Any FreeBSD 15.1 / OPNsense 26.7 guest under QEMU/KVM with Hyper-V enlightenments enabled (`hv_time` etc. — the default in most libvirt-based stacks) should show it in seconds:

```
sysctl kern.timecounter.choice                 # Hyper-V entries missing?
sysctl kern.eventtimer.et.LAPIC.frequency      # absurdly low vs. expected APIC bus clock?
vmstat -i | grep cpu0:timer                    # rate in the tens of thousands?
```

Caveats: n=1 environment (Ivy Bridge host, HPE VME/libvirt); severity is plausibly hardware-dependent — faster hosts may eat the vmexit tax without user-visible symptoms, which would explain why reports are scarce so far. Happy to provide full sysctl dumps, dmesg, or run additional diagnostics for the team.

## Ask

Restore Hyper-V enlightenment / timecounter detection for KVM guests on the FreeBSD 15.1 base (or fix the LAPIC eventtimer calibration fallback when the paravirtual reference clock is absent), ideally in a 26.7.x point release. Until then, KVM users hitting a "dog slow" 26.7 upgrade can use the i8254 tunable above.

[franco]

Thanks for the writeup! Just for reference this was discussed here: https://www.reddit.com/r/opnsense/comments/1uwz7y9/comment/oxq75eu/


Cheers,
Franco

[nero355]

Please add [SOLVED] or [SOLUTION] to the title :)

/EDIT :

Oops knew I forgot something, added.

Thnx! :)

[lesterd88]

Please add [SOLVED] or [SOLUTION] to the title :)

Oops knew I forgot something, added.

[meyergru]

Perhaps noteworthy: This does not happen on all KVM-based hypervisors and not under all circumstances, so this does not look like a generic problem.

E.g., with Proxmox, you normally do not get Hyper-V enlightenments unless you specifically ask to do so via "args: -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time" in the VM configuration.

When that feature is absent, "sysctl kern.timecounter.hardware" will give you "kvmclock", which does not have this problem at all.
Even when I specifically requested the Hyper-V feature, it gave "Hyper-V-TSC", but:

Code Select Expand

# sysctl kern.eventtimer.timer
kern.eventtimer.timer: LAPIC
# sysctl kern.eventtimer.et.LAPIC
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 500015996
kern.eventtimer.et.LAPIC.flags: 7

Which seems correct and I saw no slowdown, either.

I did not get HPET detected with 26.7, so this seems to be specific to the supplied KVM flags or what the underlying hypervisor uses as defaults.

[lesterd88]

I wondered about this as well, and it makes sense that it could be flavor specific vs an all-KVM type of issue.

I'm curious what might be causing this with VME, but not sure if I'd get to the bottom of that. I may change to kvmclock if there's a reason to do so but I haven't done much else in the name of keeping things stable for a bit.

[meyergru]

That is why I answered: the title suggests a general problem, while it seems very specific (maybe you consider changing the title).

If you want to investigate, you can look at the command line that is called underneath (with Proxmox, that is possible). It will be derived from the options you choose and may show what causes it. I had to explicitely use the "args" line above to even enable it.

I do not use Hyper-V extensions because they are strictly needed only for Windows guests and when I tried, I even noticed a slightly higher CPU load with Hyper-V enabled for OpnSense.