Constant high load on idle install

[glubarnt]

Hi everyone,

I am in the process of migrating from a dd-wrt-Router to a virtualized setup with OPNsense.

I installed it three days ago with 19.7, updated today to 20.1.
As I mentioned, OPNsense runs as a VM. The hypervisor is kvm on CentOS 8. CPU is a AMD EPYC 7282.

OPNsense gets two cores and 4GB ram.
It also gets one nic via virtio, which is a bridge. Another nic is passed directly from the hypervisor to the vm via macvtap.

Directly after the installation I noticed that the cpu was at 100% constantly, which left me wondering, because the system was not doing anything.

Looking at top on the shell, I see that unbound constantly has 6-10% cpu-usage and python 6-8%.
All other processes are below 0%.
Still, CPU has 36-45% user load, 30-45% system load and 5-40% idle.

Where could that load be coming from?

Looking in the webgui, I see a command "[rand_harvestq]" hovering around 30-50% CPU, while [idle{idle: cpu0}] is at the top most of the time.

Any clues what could the problem could be here?

Any help is much appreciated.

[opnfwb]

I saw similar issues on a fresh virtualized install. In my case, I was also seeing pflog0 promiscuous enabled/disable messages spamming the logs many times per second. This seemed to be related to IPV6 unable to pull a prefix delegation on the WAN interface of the OPNsense VM.

Try disabling IPV6 on WAN and see if this clears up? If so, it's likely related to the issue I saw in my LAB.

[glubarnt]

Hi,
thanks for your tip.

To my surprise it actually helped a little.

The load goes from 0% to 50% now, according to the dashboard.
That is better, but still not great for a system not serving any clients yet

[Gary7]

Since the process rand_harvestq is using a significant amount of CPU, you could investigate changing the value of "kern.random.harvest.mask".
The default setting in OPNsense is kern.random.harvest.mask=2047

root@OPNsense:~ # sysctl kern.random.harvest
kern.random.harvest.mask_symbolic: UMA,FS_ATIME,SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
kern.random.harvest.mask_bin: 000000000011111111111
kern.random.harvest.mask: 2047

The UMA (universal memory allocator also called zone allocator) has a potentially high rate. I don't know if UMA acts any differently on a VM vs hardware.

You could determine if CPU load decreases when lowering kern.random.harvest
kern.random.harvest = 2047    OPNsense default
kern.random.harvest = 1023    don't use UMA
kern.random.harvest = 511      FreeBSD default
kern.random.harvest = 351      max throughput according to some documentation that I found

You can set it using sysctl in a shell for testing, but I found to set the value permanently, I have to use the GUI: System -> Settings -> Tunables and add kern.random.harvest.mask

Disclaimer: If any of my information is incorrect, please correct me.

Good Luck

[allebone]

What machine type and nic driver type are you passing to the vm from KVM?

[mimugmail]

I think I read somewhere that this is related to IPv6 tracking in an interface where v6 is not available, where disabling v6 solved this.

[glubarnt]

Thanks a lot for the additional input.

Since the process rand_harvestq is using a significant amount of CPU, you could investigate changing the value of "kern.random.harvest.mask".
The default setting in OPNsense is kern.random.harvest.mask=2047

root@OPNsense:~ # sysctl kern.random.harvest
kern.random.harvest.mask_symbolic: UMA,FS_ATIME,SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
kern.random.harvest.mask_bin: 000000000011111111111
kern.random.harvest.mask: 2047

The UMA (universal memory allocator also called zone allocator) has a potentially high rate. I don't know if UMA acts any differently on a VM vs hardware.

You could determine if CPU load decreases when lowering kern.random.harvest
kern.random.harvest = 2047    OPNsense default
kern.random.harvest = 1023    don't use UMA
kern.random.harvest = 511      FreeBSD default
kern.random.harvest = 351      max throughput according to some documentation that I found

You can set it using sysctl in a shell for testing, but I found to set the value permanently, I have to use the GUI: System -> Settings -> Tunables and add kern.random.harvest.mask

Disclaimer: If any of my information is incorrect, please correct me.

Good Luck

I tried the tunable and I think it also brought the load down a little bit more, but reading this comment:

What machine type and nic driver type are you passing to the vm from KVM?

I looked up what machine type I created and by accident, it is a i440FX machine, which is not what I want.
The nic gets passedthrough, driver is virtio.

Anyways, as this is not a system that serves any traffic yet, I am gonna go ahead and resetup, because migrating i440FX to PC-Q35 is nasty and I do not want to mess it up.

I will keep you posted on the status.

Thanks again for the input and ideas.

[allebone]

I use i440fx and virtio with seabios because it works the best for me after testing all the different combinations, and uses the least cpu. I think trying to use q35 would be really hard.

[glubarnt]

I agree, because of some bug, Q-35 does not seem to be possible at the moment.

I think I am gonna leave this machine running for some time and play around with the kern.random.harvest.mask.

[glubarnt]

So,
after setting kern.random.harvest.mask to 511 and observing the load of the system for a few days, I think this is solved now.
Without serving any traffic the system sits at 0%-8% load, which is fine for me.

Thanks a lot for your help.