Messages

I wondered about this as well, and it makes sense that it could be flavor specific vs an all-KVM type of issue.

I'm curious what might be causing this with VME, but not sure if I'd get to the bottom of that. I may change to kvmclock if there's a reason to do so but I haven't done much else in the name of keeping things stable for a bit.

Please add [SOLVED] or [SOLUTION] to the title :)

Oops knew I forgot something, added.

Hi all,

Sharing this here as I originally posted on the subreddit, and was pointed to the forum to share in case anyone else has the same issue. Full disclosure, Claude helped me track this down and I asked it to write this summary up of the situation:

To note, I'm using HPE VM Essentials, which is a KVM based hypervisor.

# OPNsense 26.7 (FreeBSD 15.1) on KVM: LAPIC eventtimer miscalibration causes ~65k interrupts/sec/core — full diagnosis + mitigation

**TL;DR:** After upgrading a KVM guest from 26.1 to 26.7, the VM became nearly unusable (GUI crawling, dashboard widgets failing, configd repeatedly dying on start). Root cause: FreeBSD 15.1 no longer picks up the Hyper-V paravirtual timecounters that KVM's Hyper-V enlightenments provide (14.x on an identical VM does), and the LAPIC eventtimer gets calibrated at **416 kHz** — roughly 1000x low — so one-shot mode storms at **~65,000 timer interrupts/sec/core**. Every interrupt is a vmexit; the CPU tax made everything downstream fail. Mitigation: `kern.eventtimer.timer=i8254`. Fully reproducible, all numbers below.

## Environment

- OPNsense 26.7 (fresh upgrade from 26.1, day-one GA)
- QEMU/KVM guest (HPE VM Essentials / libvirt-based), Hyper-V enlightenments enabled (stack default)
- 4 vCPU, host CPU Xeon E5-2650v2 (Ivy Bridge)
- Backup node of a CARP pair — its peer is an **identical VM on the same host still on 26.1**, which made a clean A/B comparison possible
- Integrity verified clean before blaming the kernel: `pkg check -sa` zero mismatches, `zpool status` zero errors

## Symptoms after upgrade

- GUI extremely sluggish, dashboard widgets timing out / showing "failed"
- `configctl system status` taking 8+ seconds or hanging
- configd repeatedly killed during startup (rc wrapper / socket-wait timeouts giving up before it finished initializing)
- Sustained high ambient CPU with no traffic (node is CARP BACKUP, passing nothing)

## Diagnosis

### 1. Timer interrupt storm

```
# 26.7 node (FreeBSD 15.1):
vmstat -i | grep timer
cpu0:timer   1721502843   65470
cpu1:timer   1648644578   62699
cpu2:timer   1658830161   63087
cpu3:timer   1642275851   62457
```

~65k/sec **per core**, ~260k/sec for the VM, suspiciously pinned near 2^16. The identical VM on 26.1 (FreeBSD 14.x):

```
cpu0:timer   433639790   178
cpu1:timer   240119943    99
cpu2:timer   225831754    93
cpu3:timer   221488530    91
```

### 2. Hyper-V timecounters missing on 15.1

```
# 26.1 / FreeBSD 14.x:
kern.timecounter.hardware: Hyper-V-TSC
kern.timecounter.choice: TSC-low(-100) i8254(0) ACPI-fast(900) HPET(950) Hyper-V-TSC(3000) Hyper-V(2000) dummy(-1000000)

# 26.7 / FreeBSD 15.1, same host, same VM shape, same enlightenments:
kern.timecounter.hardware: HPET
kern.timecounter.choice: TSC-low(-100) i8254(0) ACPI-fast(900) HPET(950) dummy(-1000000)
```

The Hyper-V entries aren't just losing the priority contest — they're **absent from the choice list**, i.e., the paravirtual clock support never attached on 15.1.

### 3. LAPIC eventtimer calibrated ~1000x low

```
kern.eventtimer.et.LAPIC.frequency: 416000        # 416 kHz (!)
machdep.tsc_freq: 2599903050                      # TSC calibrated correctly at 2.6 GHz
```

KVM's emulated APIC timer runs at the ~1 GHz bus clock; the kernel believes 416 kHz. Every one-shot interval it programs is ~1000x too short → interrupt storm. TSC calibration is fine — only the LAPIC calibration is broken, consistent with a codepath that lost its paravirtual reference clock.

Supporting data point: forcing `kern.eventtimer.periodic=1` at hz=100 produced ~5,000 ints/sec/core instead of the expected ~100 — again a large constant factor too fast, consistent with the same miscalibration rather than load.

### 4. Cascade into userland

The storm's vmexit tax inflated cold Python 3.13 imports of configd's module set to ~7–8s (measured via truss + timing), exceeding the start-race patience of the rc wrapper / configctl socket-wait / dashboard polls, which then killed the half-started daemon — producing the "configd won't start" symptom. (A secondary issue on this box: the upgrade left the bytecode cache unpopulated, worsening cold imports; `python3 -m compileall` fixed that part. Note that compileall "errors" under `/usr/local/lib/python3.13/test/` are intentional bad-syntax test fixtures, not corruption.)

## Mitigation (validated)

System → Settings → Tunables: **`kern.eventtimer.timer = i8254`**

The i8254 PIT has an architecturally fixed frequency, so it can't be miscalibrated. Post-change and post-reboot:

```
irq0: attimer0   34196   395      # ~400/sec total for the whole VM (4×hz via IPI forwarding)
cpu0:timer — zero interrupts since boot
time configctl system status → 0.18s   (was 8.4s)
configd running supervised, PID 596, wins its start race cold
```

Interrupt load reduced ~650x; the box is completely normal again.

(HPET is not an option here — QEMU's HPET exposes no FSB/MSI interrupt path FreeBSD accepts, so it registers only as a timecounter, not an eventtimer. Eventtimer choices on this guest are LAPIC(600) i8254(100) RTC(0).)

## Reproduction

Any FreeBSD 15.1 / OPNsense 26.7 guest under QEMU/KVM with Hyper-V enlightenments enabled (`hv_time` etc. — the default in most libvirt-based stacks) should show it in seconds:

```
sysctl kern.timecounter.choice                 # Hyper-V entries missing?
sysctl kern.eventtimer.et.LAPIC.frequency      # absurdly low vs. expected APIC bus clock?
vmstat -i | grep cpu0:timer                    # rate in the tens of thousands?
```

Caveats: n=1 environment (Ivy Bridge host, HPE VME/libvirt); severity is plausibly hardware-dependent — faster hosts may eat the vmexit tax without user-visible symptoms, which would explain why reports are scarce so far. Happy to provide full sysctl dumps, dmesg, or run additional diagnostics for the team.

## Ask

Restore Hyper-V enlightenment / timecounter detection for KVM guests on the FreeBSD 15.1 base (or fix the LAPIC eventtimer calibration fallback when the paravirtual reference clock is absent), ideally in a 26.7.x point release. Until then, KVM users hitting a "dog slow" 26.7 upgrade can use the i8254 tunable above.