High CPU after upgrade from 25.7.9_7 to 26.1.2

Started by ck555, Today at 09:25:38 AM

Previous topic - Next topic
Hi all,

I have upgraded my Opensense running 25.7.9_7 VM to any new version currently it was running 26.1.2 and finding high CPU after a few hours. Looking at TOP all the CPU is being used by [kernel{if_io_tqg}] with all 4 cores at over 90%. I have disabled all netmap services i.e. Zenarmor and Suricata which doesn't resolve the issue. I am using CrowndSec also but didn't disable that.

No changes are made to the VM just anything newer than 25.7.9_7 has this issue. I have disabled hostmon which I saw has been known to cause high CPU. Reverting the snapshot back to 25.7.9_7 resolves the issue.

Any example of top after a few hours. At this stage its not too bad but left over night and all the CPUs hit 100%. Not sure if any other logs would be useful.

Thanks in advanced.

    0 root        -60    -     0B  1632K -        0  65:08  63.83% [kernel{if_io_tqg_0}]
System going down IMMEDIATELY                                                  o_tqg_1}]
    0 root        -60    -     0B  1632K -        3  60:38  53.24% [kernel{if_io_tqg_3}]
                                                                               cpu2}]
    0 root        -60    -     0B  1632K -        2  50:03  46.43% [kernel{if_io_tqg_2}]
   11 root        187 ki31     0B    64K RUN      3 127:42  46.25% [idle{idle: cpu3}]
   11 root        187 ki31     0B    64K CPU0     0 123:24  35.58% [idle{idle: cpu0}]
   11 root        187 ki31     0B    64K RUN      1 116:56  28.87% [idle{idle: cpu1}]
    0 root        -60    -     0B  1632K -        1  18:05  11.06% [kernel{wg_tqg_1}]
    0 root        -60    -     0B  1632K -        2   9:03   4.58% [kernel{wg_tqg_2}]
73606 root         20    0   103M    63M nanslp   3   1:01   0.48% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_watcher.php interface routes
    0 root        -60    -     0B  1632K -        3   0:47   0.20% [kernel{wg_tqg_3}]
    0 root        -60    -     0B  1632K -        0   0:32   0.15% [kernel{wg_tqg_0}]
16388 root         20    0    49M    15M kqread   1   0:00   0.09% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid{syslog
47389 root         20    0    17M  5500K CPU3     3   0:00   0.07% top -aSH
10987 root         20    0    14M  2916K bpf      2   0:08   0.06% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
    2 root        -60    -     0B    64K WAIT     3   0:05   0.05% [clock{clock (0)}]
32543 root         20    0    40M    20M select   1   0:04   0.05% /usr/local/sbin/snmpd -p /var/run/net_snmpd.pid
16285 root         20    0    29M  7876K select   3   0:06   0.04% /usr/local/bin/vmtoolsd -c /usr/local/share/vmware-tools/tools.conf -p /usr/local/lib/open-
86903 root         20    0    20M  9160K select   0   0:00   0.03% sshd-session: root@pts/0 (sshd-session)
    7 root        -16    -     0B    16K pftm     3   0:04   0.02% [pf purge]
16388 root         20    0    49M    15M kqread   2   0:02   0.01% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid{syslog
11212 root         20    0    27M    14M select   1   0:01   0.01% /usr/local/bin/python3 /usr/local/sbin/configctl -e -t 0.5 system event config_changed (pyt
  687 root         20    0    15M  4268K select   1   0:01   0.01% /sbin/devd
    9 root        -16    -     0B    48K psleep   3   0:04   0.01% [pagedaemon{dom0}]
15601 root         20    0    28M    15M select   3   0:01   0.01% /usr/local/bin/python3 /usr/local/opnsense/scripts/syslog/lockout_handler (python3.11)
   12 root        -64    -     0B   720K WAIT     3   0:00   0.01% [intr{irq14: ata0}]
14402 root         20    0    24M  8100K select   2   0:01   0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf{ntpd}
    0 root        -16    -     0B  1632K -        2   0:00   0.01% [kernel{z_rd_int_1}]
    8 root        -16    -     0B    16K -        2   0:01   0.01% [rand_harvestq]
95698 root         20    0    14M  2692K nanslp   3   0:00   0.01% /usr/local/bin/dpinger -f -S -r 0 -i WAN_DHCP6 -B fe80::250:56ff:fe9f:bfcf%vmx1 -p /var/run
72018 root         20    0    14M  2560K kqread   3   0:00   0.00% /usr/sbin/rtsold -aiu -p /var/run/rtsold.pid -A /var/etc/rtsold_script.sh -R /usr/local/opn
51456 root         20    0  1312M   108M uwait    2   0:01   0.00% /usr/local/AdGuardHome/AdGuardHome -s run{AdGuardHome}
51456 root         20    0  1312M   108M kqread   2   0:02   0.00% /usr/local/AdGuardHome/AdGuardHome -s run{AdGuardHome}
95698 root         20    0    14M  2692K accept   3   0:00   0.00% /usr/local/bin/dpinger -f -S -r 0 -i WAN_DHCP6 -B fe80::250:56ff:fe9f:bfcf%vmx1 -p /var/run
74497 root         20    0    14M  2636K select   3   0:00   0.00% rtsold: system.syslog (rtsold)
   12 root        -60    -     0B   720K WAIT     3   0:01   0.00% [intr{swi1: netisr 0}]
57839 root         20    0    18M  2728K accept   3   0:00   0.00% /usr/local/bin/dpinger -f -S -r 0 -i WAN_DHCP -B 202.153.211.4 -p /var/run/dpinger_WAN_DHCP
    1 root         32    0    12M  1084K wait     3   0:00   0.00% [init]
51962 root         20    0    26M    13M kqread   3   0:02   0.00% /usr/local/sbin/lighttpd -f /usr/local/etc/lighttpd_webgui/lighttpd.conf
50253 nobody       20    0    15M  3364K select   3   0:00   0.00% /usr/local/sbin/dnsmasq -x /var/run/dnsmasq.pid -C /usr/local/etc/dnsmasq.conf
   12 root        -60    -     0B   720K WAIT     2   0:00   0.00% [intr{swi1: pfsync}]
70870 _dhcp        20    0    14M  2832K select   3   0:00   0.00% dhclient: vmx1 (dhclient)
16285 root         20    0    29M  7876K uwait    2   0:00   0.00% /usr/local/bin/vmtoolsd -c /usr/local/share/vmware-tools/tools.conf -p /usr/local/lib/open-
    4 root        -16    -     0B    48K -        1   0:01   0.00% [cam{doneq0}]
13474 root         20    0    21M  9024K select   2   0:00   0.00% /usr/local/bin/monit -c /usr/local/etc/monitrc{monit}
51456 root         20    0  1312M   108M uwait    1   0:01   0.00% /usr/local/AdGuardHome/AdGuardHome -s run{AdGuardHome}
    2 root        -60    -     0B    64K WAIT     2   0:00   0.00% [clock{clock (2)}]
    2 root        -60    -     0B    64K WAIT     1   0:00   0.00% [clock{clock (1)}]
    2 root        -60    -     0B    64K WAIT     3   0:00   0.00% [clock{clock (3)}]
   16 root         20    -     0B   144K sdflus   3   0:00   0.00% [bufdaemon{/usr/local/zenarmor}]
    6 root         -8    -     0B   896K dbuf_e   0   0:01   0.00% [zfskern{dbuf_evict_thread}]
95698 root         20    0    14M  2692K sbwait   3   0:00   0.00% /usr/local/bin/dpinger -f -S -r 0 -i WAN_DHCP6 -B fe80::250:56ff:fe9f:bfcf%vmx1 -p /var/run
    6 root         -8    -     0B   896K mmp->m   2   0:00   0.00% [zfskern{mmp_thread_enter}]
    6 root         -8    -     0B   896K t->zth   0   0:00   0.00% [zfskern{arc_reap}]
    6 root         -8    -     0B   896K t->zth   1   0:00   0.00% [zfskern{arc_evict}]
   16 root        -16    -     0B   144K psleep   2   0:00   0.00% [bufdaemon{bufdaemon}]