High CPU Utilization (Interrupts) with XCP-ng with Fix Provided

[HamiltonWDS]

I had encountered this same problem with Version 24.7, and seeing that there was Version 25.1 in Beta, I thought to see if this problem was resolved.
Unfortunately that is not the case as the problem remains.
Fortunately there is a fix for it, which I will provide the steps below.

Problem:
When using OPNsense v25.1 (and v24.7) as a virtual machine within XCP-ng (and Xenserver) there is a problem of false reporting by FreeBSD v14 of Interrupts to Xen. This then causes health monitor checks with the CPU as reading high utilization while at idle (high 50%). When at the OPNsense Dashboard, the CPU widget indicates high usage, as well as, along with the Netdata plugin if used. Installing the Xen Tools plugin has no effect.
However, the hypervisor indicates minimal CPU utilization. Additionally Bandwidth performance is not impacted.

Mitigation:
Applying a patch and rebuilding the kernel from sources solves the problem. For long term purposes, is to recommend this patch corrections to the Xen files into the development repository for testing. If acceptable to be part of the production build.
So far after applying this patch have not experienced further problems.
This also worked for both versions of 24.7 and 25.1.

Patch Reference: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277231
Actual Patch: https://bz-attachments.freebsd.org/attachment.cgi?id=248948

Steps for Applying the Patch:
NOTE: These are the steps I took. For actual patching, many steps can be omitted.
Obtain the Patch
1. Go to:   https://bz-attachments.freebsd.org/attachment.cgi?id=248948
2. Copy the entire patch page as is.
2a. Select All (CTRL-A) and Copy (CTRL-C)

Implement Patch
1. Once OPNsense is ready, open PUTTY for SSH connection and login.
2. At the OPNsense Shell, type in "8" for Shell and hit Enter.
Perform Verifications:
3. Verify Interrupt usage:

Code Select Expand

top -P4. The top portion of the screen should indicate high readings of Interrupts.
4b. Hit CTRL-C to cancel the Top Monitoring.
5. Verify the current Kernel:

Code Select Expand

uname -aExample:

Code Select Expand

FreeBSD OPNsense.localdomain 14.2-RELEASE FreeBSD 14.2-RELEASE stable/25.1-n269579-cd5d25393d6 SMP amd646. Verify Disk Space Usage before patching.
6c. Check how much disk space is being used by build files:

Code Select Expand

du -h /usr/obj
du -h /usr/src6d. Make a note of the size, for example 512B (empty). Basically, should be empty.

Create the Patch File:
7. Navigate to a temporary directory

Code Select Expand

cd /tmp8. Create a new file called 277231.patch

Code Select Expand

vi 277231.patch9. Paste the entire patch content as-is:
9a. Press "i" to enter insert mode.
9b. Paste the patch content (right click only).
9c. Press Esc to exit insert mode.
9d. Type ":wq" to save and exit.

Apply the Patch to the Source Code:
10. Navigate to the source directory where the affected files are located:

Code Select Expand

cd /usr/src11. Download the OPNsense source code:

Code Select Expand

opnsense-code src12. After the source code has been transferred, verify that the required files now exist:

Code Select Expand

find /usr/src -name xen_intr.c
find /usr/src -name xen_arch_intr.c13. Apply the patch:

Code Select Expand

patch -p1 < /tmp/277231.patch
Build the New Kernel:
14. Create the OPNsense Kernel Config File, by copying the GENERIC kernel config as a starting point:

Code Select Expand

cp /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/OPNsense15. Rebuild the Kernel:

Code Select Expand

make buildkernel KERNCONF=OPNsenseNOTE: This will take some time to complete, approximately 35 minutes if using x12 CPUs and 12GB RAM.
16. If the build completes successfully, should see something like:

Code Select Expand

>>> Kernel build for OPNsense completed on Sat Jan 6 12:34:56 UTC 2025
Install the New Kernel:
17. After the build completes, install the new kernel:

Code Select Expand

make installkernel KERNCONF=OPNsenseNOTE: This will take some time, but much less than the previous build.
18. Once the kernel is installed, reboot the system:

Code Select Expand

reboot
Verify the New Kernel
1. After OPNsense comes back online, log in via the Web GUI.
2. The CPU Widget should now appear with low CPU Utilization.
3. Open up a SSH connection and login.
4. Verify that OPNsense is running the new kernel:

Code Select Expand

uname -aExample:

Code Select Expand

FreeBSD OPNsense.localdomain 14.2-RELEASE FreeBSD 14.2-RELEASE #0 stable/25.1-n269581-b4fca65e6b8-dirty: Sun Jan  5 02:41:21 UTC 20255. Verify that the interrupts issue is resolved by running:

Code Select Expand

top -P6. Check the intr column to ensure that interrupt usage has decreased.

Clean Up Build Artifacts
With the new build completed, now it is time to clean up.
1. Run the following commands:

Code Select Expand

cd /usr/src
make clean
rm -rf /usr/obj/*
rm -rf /usr/src2. Verify Disk Space Usage

Code Select Expand

du -h /usr/obj
du -h /usr/src3. Reclaim Space from Old Kernel Files, by running the following commands:
3a. Check the size of old kernels:

Code Select Expand

ls -lh /boot/kernel.oldIf there is a return of files, then perform the following command to remove them:

Code Select Expand

rm -rf /boot/kernel.old
That completes the steps for the patching.


Of note, I tested v25.1 directly on the hardward and did not have this issue. Appears to be only with Xenserver related systems.