"
Thanks. This does not look like the LAPIC calibration problem from the thread I linked.
Your LAPIC frequency of about 500 MHz looks sane and is almost exactly what was reported for the working Proxmox case there. The broken case had a frequency off by roughly three orders of magnitude and around 65k timer interrupts/sec on every vCPU.
The `vmstat -i` numbers are interesting, but note that the displayed rate is averaged since boot, so it may hide what happens during one of the short stalls.
Since you are using `kvmclock`, I think one simple A/B test would still be worthwhile:
Leave everything else unchanged and see whether the stalls still occur. With several events per hour it should not take too long to get a useful result.
You can switch back with:
I would not conclude from the DTrace samples yet that the TCP retransmission timers are the cause. They may also be a consequence of the actual stall: if packet processing stops briefly, retransmission timers expire and `softclock_thread` subsequently has a lot of work to do.
The snapshot-related KVM clock problem I mentioned earlier also looks less likely in your case, since you have many events which clearly do not coincide with snapshots or backups.
Your LAPIC frequency of about 500 MHz looks sane and is almost exactly what was reported for the working Proxmox case there. The broken case had a frequency off by roughly three orders of magnitude and around 65k timer interrupts/sec on every vCPU.
The `vmstat -i` numbers are interesting, but note that the displayed rate is averaged since boot, so it may hide what happens during one of the short stalls.
Since you are using `kvmclock`, I think one simple A/B test would still be worthwhile:
Code Select
sysctl kern.timecounter.hardware=ACPI-fast
Leave everything else unchanged and see whether the stalls still occur. With several events per hour it should not take too long to get a useful result.
You can switch back with:
Code Select
sysctl kern.timecounter.hardware=kvmclock
I would not conclude from the DTrace samples yet that the TCP retransmission timers are the cause. They may also be a consequence of the actual stall: if packet processing stops briefly, retransmission timers expire and `softclock_thread` subsequently has a lot of work to do.
The snapshot-related KVM clock problem I mentioned earlier also looks less likely in your case, since you have many events which clearly do not coincide with snapshots or backups.
