See the two attached graphs. The gap in the middle is patching from 23.1 to 23.7. There are a lot of ISRs popping off. Throughput is slower. Speed test are affecting the CPU much more than in the past.
Graph 1 shows the CPU is running hotter.
Graph 2 shows the higher ISRs.
We're getting FreeBSD 13.2-RELEASE-p2 in 23.7.1 if I didn't misread something so I'd redo these checks after
For Info, J3455 CPU.
Hopping on this thread to report asinine CPU usage in the gateway_watcher.php script post-upgrade to 23.7. Configuration did not change from 23.1.There's absolutely no reason this should be using orders of magnitude more CPU time than suricata, ever. Both primary and backup systems show the same behavior.
gateway_watcher.php is new in 23.7 and it practically runs forever for technical reasons. I'm sure sometimes it needs CPU especially in the gateway alert cases but I can't imagine it's doing much in between.
Cheers,
Franco
PS: The script is here... https://github.com/opnsense/core/blob/master/src/opnsense/scripts/routes/gateway_watcher.php
Perhaps reloading the config is heavier than expected, but it would also depend on the the size of the config.xml. Additional pairs of eyes are appreciated.
Here's what I'm seeing:
On an APU4 where there's a single GW monitoring 1.1.1.1 I see a constant CPU activity averaging between 2.8-3%, another APU has it around 3-4% and yet another one doesn't even have it running.
N5105 - avg 2%
yea on Intel N200, i am seeing 4-8% constant cpu usage. keeping my box elevated 2-3w of "idle", 23.1.11 was 11w idle, now is 13.7-14.5w
last pid: 6023; load averages: 0.46, 0.40, 0.42 up 0+06:48:37 15:59:26
444 threads: 4 running, 418 sleeping, 22 waiting
CPU 0: 9.9% user, 0.0% nice, 1.7% system, 1.7% interrupt, 86.8% idle
CPU 1: 3.3% user, 0.0% nice, 1.7% system, 0.0% interrupt, 95.0% idle
CPU 2: 0.0% user, 0.0% nice, 1.7% system, 0.0% interrupt, 98.3% idle
CPU 3: 0.0% user, 0.0% nice, 2.5% system, 0.0% interrupt, 97.5% idle
Mem: 1189M Active, 4723M Inact, 6457M Wired, 156K Buf, 3288M Free
ARC: 1828M Total, 131M MFU, 1585M MRU, 521K Anon, 14M Header, 97M Other
1515M Compressed, 5339M Uncompressed, 3.52:1 Ratio
Swap: 8192M Total, 8192M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND
2612 root 49 0 79M 50M nanslp 2 33:24 9.12% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_watcher.php interface routes alarm
Quote from: franco on August 08, 2023, 09:40:51 PM
Perhaps reloading the config is heavier than expected, but it would also depend on the the size of the config.xml. Additional pairs of eyes are appreciated.
Not really seeing the same problem here (CPU-usage wise, but it is the one with the most CPU time racked up), but I profiled it with xdebug and other than sleep the top two calls are:
- return_gateways_status, with most of the time spent in legacy_interfaces_details/php::exec
- parse_config, with almost all of the time spent in OPNsense\Core\Config->toArray (lots of recursive calls for this one, sounds like it is serializing XML into a PHP array?)
Perhaps a caching option, checking mtime to decide whether to reload?
I am also experiencing hotter temps on the CPU now after the upgrade
OPNsense 23.7.1_3-amd64
FreeBSD 13.2-RELEASE-p2
OpenSSL 1.1.1v 1 Aug 2023
CPU type 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz (4 cores, 8 threads)
Getting the following Error Reports Since Upgrade:
System Information:
User-Agent Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0
FreeBSD 13.2-RELEASE-p2 stable/23.7-n254746-cdad4e9df7f SMP amd64
OPNsense 23.7.1_3 239f8d1f8
Plugins os-crowdsec-1.0.6 os-ddclient-1.14 os-dmidecode-1.1_1 os-dyndns-1.27_3 os-etpro-telemetry-1.6_1 os-intrusion-detection-content-et-open-1.0.1 os-intrusion-detection-content-et-pro-1.0.2_1 os-intrusion-detection-content-pt-open-1.0_1 os-intrusion-detection-content-snort-vrt-1.1_1 os-iperf-1.0_1 os-maltrail-1.10 os-netdata-1.2_1 os-nginx-1.32.1_3 os-sensei-1.14.2 os-sensei-agent-1.14 os-sensei-updater-1.14 os-sunnyvalley-1.2_3 os-telegraf-1.12.8 os-theme-cicada-1.34 os-theme-rebellion-1.8.8 os-vnstat-1.3_1 os-wireguard-1.13_7
Time Wed, 09 Aug 2023 13:26:41 +0000
OpenSSL 1.1.1v 1 Aug 2023
Python 3.9.17
PHP 8.2.8
PHP Errors:
[09-Aug-2023 12:27:09 Etc/UTC] Error: Call to undefined function lock() in /usr/local/etc/inc/plugins.inc.d/dyndns/phpDynDNS.inc:158
Stack trace:
#0 /usr/local/etc/inc/plugins.inc.d/dyndns.inc(169): updatedns->__construct('dyndns', 'gambit.dnsdojo....', 'secdoc', 'b6f0e74070da11e...', NULL, '', 'wan', NULL, NULL, NULL, '', NULL, '', '', '', '', 'wan', '0', false, false, false)
#1 /usr/local/etc/inc/plugins.inc.d/dyndns.inc(212): dyndns_configure_client(Array)
#2 /usr/local/etc/inc/plugins.inc(304): dyndns_configure_do(true)
#3 /usr/local/etc/rc.bootup(104): plugins_configure('bootup', true)
#4 {main}
[09-Aug-2023 12:27:10 Etc/UTC] Error: Call to undefined function lock() in /usr/local/etc/inc/plugins.inc.d/dyndns/phpDynDNS.inc:158
Stack trace:
#0 /usr/local/etc/inc/plugins.inc.d/dyndns.inc(169): updatedns->__construct('dyndns', 'gambit.dnsdojo....', 'secdoc', 'b6f0e74070da11e...', NULL, '', 'wan', NULL, NULL, NULL, '', NULL, '', '', '', '', 'wan', '0', false, false, false)
#1 /usr/local/etc/inc/plugins.inc.d/dyndns.inc(212): dyndns_configure_client(Array)
#2 /usr/local/etc/inc/plugins.inc(304): dyndns_configure_do(false, 'wan')
#3 /usr/local/etc/rc.newwanip(174): plugins_configure('newwanip', false, Array)
#4 {main}
dmesg.boot:
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.2-RELEASE-p2 stable/23.7-n254746-cdad4e9df7f SMP amd64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
VT(efifb): resolution 1366x768
CPU: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz (2803.20-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x806c1 Family=0x6 Model=0x8c Stepping=1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0xf3bfa7eb<FSGSBASE,TSCADJ,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,SHA,AVX512BW,AVX512VL>
Structured Extended Features2=0x18c05fde<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,AVX512VPOPCNTDQ,RDPID,MOVDIRI,MOVDIR64B>
Structured Extended Features3=0xfc100510<FSRM,AVX512VP2INTERSECT,MD_CLEAR,IBT,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
IA32_ARCH_CAPS=0x6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
real memory = 34359738368 (32768 MB)
avail memory = 33032052736 (31501 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 1 5 4 6 7 2 3
random: entropy device external interface
wlan: mac acl policy registered
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x43ca5000-0x43ca501e
smbios0: Version: 3.3, BCD Revision: 3.3
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <ALASKA A M I >
ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - ^SPI1.SPFD.CVFD (20201113/dspkginit-605)
ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - ^SPI1.SPFD.CVFD (20201113/dspkginit-605)
ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - ^XHCI.RHUB.HS04.VI2C (20201113/dspkginit-605)
ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - ^XHCI.RHUB.HS04.VI2C (20201113/dspkginit-605)
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 19200000 Hz quality 950
Event timer "HPET" frequency 19200000 Hz quality 550
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x3000-0x303f mem 0x6000000000-0x6000ffffff,0x4000000000-0x400fffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
pcib1: <ACPI PCI-PCI bridge> irq 19 at device 6.0 on pci0
pci1: <ACPI PCI bus> on pcib1
nvme0: <Generic NVMe Device> mem 0x51600000-0x51603fff,0x51604000-0x516040ff irq 16 at device 0.0 on pci1
xhci0: <Intel Tiger Lake-LP Thunderbolt 4 USB controller> mem 0x6001110000-0x600111ffff at device 13.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
xhci1: <Intel Tiger Lake-LP USB 3.2 controller> mem 0x6001100000-0x600110ffff irq 16 at device 20.0 on pci0
xhci1: 32 bytes context size, 64-bit DMA
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <AHCI SATA controller> port 0x3090-0x3097,0x3080-0x3083,0x3060-0x307f mem 0x51700000-0x51701fff,0x51703000-0x517030ff,0x51702000-0x517027ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> irq 19 at device 3.0 on pci3
pci4: <PCI bus> on pcib4
igc0: <Intel(R) Ethernet Controller I226-V> mem 0x51400000-0x514fffff,0x51500000-0x51503fff irq 19 at device 0.0 on pci4
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 4 RX queues 4 TX queues
igc0: Using MSI-X interrupts with 5 vectors
igc0: Ethernet address: 00:90:27:e8:33:1f
igc0: netmap queues/slots: TX 4/1024, RX 4/1024
pcib5: <PCI-PCI bridge> irq 19 at device 7.0 on pci3
pci5: <PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
igc1: <Intel(R) Ethernet Controller I226-V> mem 0x51100000-0x511fffff,0x51200000-0x51203fff irq 17 at device 0.0 on pci6
igc1: Using 1024 TX descriptors and 1024 RX descriptors
igc1: Using 4 RX queues 4 TX queues
igc1: Using MSI-X interrupts with 5 vectors
igc1: Ethernet address: 00:90:27:e8:33:20
igc1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib7: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci7: <ACPI PCI bus> on pcib7
igc2: <Intel(R) Ethernet Controller I226-V> mem 0x50e00000-0x50efffff,0x50f00000-0x50f03fff irq 18 at device 0.0 on pci7
igc2: Using 1024 TX descriptors and 1024 RX descriptors
igc2: Using 4 RX queues 4 TX queues
igc2: Using MSI-X interrupts with 5 vectors
igc2: Ethernet address: 00:90:27:e8:33:21
igc2: netmap queues/slots: TX 4/1024, RX 4/1024
pcib8: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci8: <ACPI PCI bus> on pcib8
igc3: <Intel(R) Ethernet Controller I226-V> mem 0x50b00000-0x50bfffff,0x50c00000-0x50c03fff irq 19 at device 0.0 on pci8
igc3: Using 1024 TX descriptors and 1024 RX descriptors
igc3: Using 4 RX queues 4 TX queues
igc3: Using MSI-X interrupts with 5 vectors
igc3: Ethernet address: 00:90:27:e8:33:22
igc3: netmap queues/slots: TX 4/1024, RX 4/1024
pcib9: <ACPI PCI-PCI bridge> irq 16 at device 29.0 on pci0
pci9: <ACPI PCI bus> on pcib9
igc4: <Intel(R) Ethernet Controller I226-V> mem 0x50800000-0x508fffff,0x50900000-0x50903fff irq 16 at device 0.0 on pci9
igc4: Using 1024 TX descriptors and 1024 RX descriptors
igc4: Using 4 RX queues 4 TX queues
igc4: Using MSI-X interrupts with 5 vectors
igc4: Ethernet address: 00:90:27:e8:33:23
igc4: netmap queues/slots: TX 4/1024, RX 4/1024
pcib10: <ACPI PCI-PCI bridge> irq 17 at device 29.1 on pci0
pci10: <ACPI PCI bus> on pcib10
igc5: <Intel(R) Ethernet Controller I226-V> mem 0x50500000-0x505fffff,0x50600000-0x50603fff irq 17 at device 0.0 on pci10
igc5: Using 1024 TX descriptors and 1024 RX descriptors
igc5: Using 4 RX queues 4 TX queues
igc5: Using MSI-X interrupts with 5 vectors
igc5: Ethernet address: 00:90:27:e8:33:24
igc5: netmap queues/slots: TX 4/1024, RX 4/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
hdac0: <Intel Tiger Lake HDA Controller> mem 0x6001128000-0x600112bfff,0x6001000000-0x60010fffff irq 16 at device 31.3 on pci0
pci0: <serial bus> at device 31.5 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> on acpi0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
hwpstate_intel4: <Intel Speed Shift> on cpu4
hwpstate_intel5: <Intel Speed Shift> on cpu5
hwpstate_intel6: <Intel Speed Shift> on cpu6
hwpstate_intel7: <Intel Speed Shift> on cpu7
Timecounter "TSC-low" frequency 1401613928 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ugen0.1: <Intel XHCI root HUB> at usbus0
ugen1.1: <Intel XHCI root HUB> at usbus1
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub1 on usbus1
uhub1: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
nvme0: Allocated 200MB host memory buffer
nvd0: <WD Blue SN570 1TB> NVMe namespace
nvme0: async event occurred (type 0x1, info 0x01, page 0x02)
nvd0: 953869MB (1953525168 512 byte sectors)
hdacc0: <Intel Tiger Lake HDA CODEC> at cad 2 on hdac0
hdaa0: <Intel Tiger Lake Audio Function Group> at nid 1 on hdacc0
nvme0: temperature above threshold
pcm0: <Intel Tiger Lake (HDMI/DP 8ch)> at nid 4 on hdaa0
Trying to mount root from zfs:zroot/ROOT/default []...
uhub0: 5 ports with 5 removable, self powered
uhub1: 16 ports with 16 removable, self powered
ugen1.2: <Telink Wireless Receiver> at usbus1
ukbd0 on uhub1
ukbd0: <Telink Wireless Receiver, class 0/0, rev 1.10/1.04, addr 1> on usbus1
kbd2 at ukbd0
igc0: link state changed to UP
igc1: link state changed to UP
ichsmb0: <Intel Tiger Lake SMBus controller> port 0xefa0-0xefbf mem 0x6001130000-0x60011300ff irq 16 at device 31.4 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_wmi0: <ACPI-WMI mapping> on acpi0
acpi_wmi0: cannot find EC device
acpi_wmi0: Embedded MOF found
ACPI: \134_SB.WFDE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20201113/nsarguments-361)
acpi_wmi1: <ACPI-WMI mapping> on acpi0
acpi_wmi1: cannot find EC device
acpi_wmi1: Embedded MOF found
ACPI: \134_SB.WFTE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20201113/nsarguments-361)
ums0 on uhub1
ums0: <Telink Wireless Receiver, class 0/0, rev 1.10/1.04, addr 1> on usbus1
ums0: 5 buttons and [XYZ] coordinates ID=1
lo0: link state changed to UP
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
pflog0: permanently promiscuous mode enabled
igc1: link state changed to DOWN
vlan0: changing name to 'vlan01'
vlan1: changing name to 'vlan02'
vlan2: changing name to 'vlan04'
vlan3: changing name to 'vlan040'
vlan4: changing name to 'vlan05'
vlan5: changing name to 'vlan06'
vlan6: changing name to 'vlan07'
vlan7: changing name to 'vlan08'
vlan8: changing name to 'vlan09'
[fib_algo] inet.0 (bsearch4#16) rebuild_fd_flm: switching algo to radix4_lockless
igc0: link state changed to DOWN
igc1: link state changed to UP
vlan09: link state changed to UP
vlan08: link state changed to UP
vlan01: link state changed to UP
vlan040: link state changed to UP
vlan04: link state changed to UP
vlan02: link state changed to UP
igc0: link state changed to UP
WARNING: attempt to domain_add(netgraph) after domainfinalize()
935.338437 [1173] generic_netmap_attach Emulated adapter for vlan09 created (prev was NULL)
935.338458 [1078] generic_netmap_dtor Emulated netmap adapter for vlan09 destroyed
935.338546 [1173] generic_netmap_attach Emulated adapter for vlan09 created (prev was NULL)
935.666059 [ 321] generic_netmap_register Emulated adapter for vlan09 activated
935.666146 [ 851] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048
935.666274 [ 851] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048
935.801747 [1173] generic_netmap_attach Emulated adapter for vlan040 created (prev was NULL)
935.801766 [1078] generic_netmap_dtor Emulated netmap adapter for vlan040 destroyed
935.801857 [1173] generic_netmap_attach Emulated adapter for vlan040 created (prev was NULL)
935.801954 [ 321] generic_netmap_register Emulated adapter for vlan040 activated
935.808541 [1173] generic_netmap_attach Emulated adapter for vlan05 created (prev was NULL)
935.808562 [1078] generic_netmap_dtor Emulated netmap adapter for vlan05 destroyed
935.808651 [1173] generic_netmap_attach Emulated adapter for vlan05 created (prev was NULL)
935.808737 [ 321] generic_netmap_register Emulated adapter for vlan05 activated
935.813452 [1173] generic_netmap_attach Emulated adapter for vlan07 created (prev was NULL)
935.813470 [1078] generic_netmap_dtor Emulated netmap adapter for vlan07 destroyed
935.813558 [1173] generic_netmap_attach Emulated adapter for vlan07 created (prev was NULL)
935.813648 [ 321] generic_netmap_register Emulated adapter for vlan07 activated
935.820242 [ 851] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048
935.820332 [ 851] iflib_netmap_config txr 4 rxr 4 txd 1024 rxd 1024 rbufsz 2048
935.958708 [1173] generic_netmap_attach Emulated adapter for vlan06 created (prev was NULL)
935.958728 [1078] generic_netmap_dtor Emulated netmap adapter for vlan06 destroyed
935.958821 [1173] generic_netmap_attach Emulated adapter for vlan06 created (prev was NULL)
935.958918 [ 321] generic_netmap_register Emulated adapter for vlan06 activated
935.963221 [1173] generic_netmap_attach Emulated adapter for vlan02 created (prev was NULL)
935.963241 [1078] generic_netmap_dtor Emulated netmap adapter for vlan02 destroyed
935.963332 [1173] generic_netmap_attach Emulated adapter for vlan02 created (prev was NULL)
935.963427 [ 321] generic_netmap_register Emulated adapter for vlan02 activated
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
igc0: promiscuous mode enabled
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp1: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp0: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp0: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp1: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp5: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp6: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp7: critical temperature detected, suggest system shutdown
coretemp3: critical temperature detected, suggest system shutdown
coretemp0: critical temperature detected, suggest system shutdown
959.843286 [ 296] generic_netmap_unregister Emulated adapter for vlan06 deactivated
959.847697 [1078] generic_netmap_dtor Emulated netmap adapter for vlan06 destroyed
959.849756 [ 296] generic_netmap_unregister Emulated adapter for vlan040 deactivated
959.849818 [1078] generic_netmap_dtor Emulated netmap adapter for vlan040 destroyed
959.954261 [ 296] generic_netmap_unregister Emulated adapter for vlan05 deactivated
959.954313 [1078] generic_netmap_dtor Emulated netmap adapter for vlan05 destroyed
959.956177 [ 296] generic_netmap_unregister Emulated adapter for vlan07 deactivated
959.956235 [1078] generic_netmap_dtor Emulated netmap adapter for vlan07 destroyed
959.958255 [ 296] generic_netmap_unregister Emulated adapter for vlan02 deactivated
959.958321 [1078] generic_netmap_dtor Emulated netmap adapter for vlan02 destroyed
959.960420 [ 296] generic_netmap_unregister Emulated adapter for vlan09 deactivated
959.961771 [1078] generic_netmap_dtor Emulated netmap adapter for vlan09 destroyed
igc0: promiscuous mode disabled
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 2 0 0 0 0 0 done
All buffers synced.
Uptime: 20h34m46s
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 1 3 2 7 6 5 4
random: entropy device external interface
wlan: mac acl policy registered
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x43ca5000-0x43ca501e
smbios0: Version: 3.3, BCD Revision: 3.3
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <ALASKA A M I >
ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - ^SPI1.SPFD.CVFD (20201113/dspkginit-605)
ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - ^SPI1.SPFD.CVFD (20201113/dspkginit-605)
ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - ^XHCI.RHUB.HS04.VI2C (20201113/dspkginit-605)
ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - ^XHCI.RHUB.HS04.VI2C (20201113/dspkginit-605)
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 19200000 Hz quality 950
Event timer "HPET" frequency 19200000 Hz quality 550
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x3000-0x303f mem 0x6000000000-0x6000ffffff,0x4000000000-0x400fffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
pcib1: <ACPI PCI-PCI bridge> irq 19 at device 6.0 on pci0
pci1: <ACPI PCI bus> on pcib1
nvme0: <Generic NVMe Device> mem 0x51600000-0x51603fff,0x51604000-0x516040ff irq 16 at device 0.0 on pci1
xhci0: <Intel Tiger Lake-LP Thunderbolt 4 USB controller> mem 0x6001110000-0x600111ffff at device 13.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
xhci1: <Intel Tiger Lake-LP USB 3.2 controller> mem 0x6001100000-0x600110ffff irq 16 at device 20.0 on pci0
xhci1: 32 bytes context size, 64-bit DMA
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <AHCI SATA controller> port 0x3090-0x3097,0x3080-0x3083,0x3060-0x307f mem 0x51700000-0x51701fff,0x51703000-0x517030ff,0x51702000-0x517027ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> irq 19 at device 3.0 on pci3
pci4: <PCI bus> on pcib4
igc0: <Intel(R) Ethernet Controller I226-V> mem 0x51400000-0x514fffff,0x51500000-0x51503fff irq 19 at device 0.0 on pci4
igc0: Using 1024 TX descriptors and 1024 RX descriptors
igc0: Using 4 RX queues 4 TX queues
igc0: Using MSI-X interrupts with 5 vectors
igc0: Ethernet address: 00:90:27:e8:33:1f
igc0: netmap queues/slots: TX 4/1024, RX 4/1024
pcib5: <PCI-PCI bridge> irq 19 at device 7.0 on pci3
pci5: <PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
igc1: <Intel(R) Ethernet Controller I226-V> mem 0x51100000-0x511fffff,0x51200000-0x51203fff irq 17 at device 0.0 on pci6
igc1: Using 1024 TX descriptors and 1024 RX descriptors
igc1: Using 4 RX queues 4 TX queues
igc1: Using MSI-X interrupts with 5 vectors
igc1: Ethernet address: 00:90:27:e8:33:20
igc1: netmap queues/slots: TX 4/1024, RX 4/1024
pcib7: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci7: <ACPI PCI bus> on pcib7
igc2: <Intel(R) Ethernet Controller I226-V> mem 0x50e00000-0x50efffff,0x50f00000-0x50f03fff irq 18 at device 0.0 on pci7
igc2: Using 1024 TX descriptors and 1024 RX descriptors
igc2: Using 4 RX queues 4 TX queues
igc2: Using MSI-X interrupts with 5 vectors
igc2: Ethernet address: 00:90:27:e8:33:21
igc2: netmap queues/slots: TX 4/1024, RX 4/1024
pcib8: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci8: <ACPI PCI bus> on pcib8
igc3: <Intel(R) Ethernet Controller I226-V> mem 0x50b00000-0x50bfffff,0x50c00000-0x50c03fff irq 19 at device 0.0 on pci8
igc3: Using 1024 TX descriptors and 1024 RX descriptors
igc3: Using 4 RX queues 4 TX queues
igc3: Using MSI-X interrupts with 5 vectors
igc3: Ethernet address: 00:90:27:e8:33:22
igc3: netmap queues/slots: TX 4/1024, RX 4/1024
pcib9: <ACPI PCI-PCI bridge> irq 16 at device 29.0 on pci0
pci9: <ACPI PCI bus> on pcib9
igc4: <Intel(R) Ethernet Controller I226-V> mem 0x50800000-0x508fffff,0x50900000-0x50903fff irq 16 at device 0.0 on pci9
igc4: Using 1024 TX descriptors and 1024 RX descriptors
igc4: Using 4 RX queues 4 TX queues
igc4: Using MSI-X interrupts with 5 vectors
igc4: Ethernet address: 00:90:27:e8:33:23
igc4: netmap queues/slots: TX 4/1024, RX 4/1024
pcib10: <ACPI PCI-PCI bridge> irq 17 at device 29.1 on pci0
pci10: <ACPI PCI bus> on pcib10
igc5: <Intel(R) Ethernet Controller I226-V> mem 0x50500000-0x505fffff,0x50600000-0x50603fff irq 17 at device 0.0 on pci10
igc5: Using 1024 TX descriptors and 1024 RX descriptors
igc5: Using 4 RX queues 4 TX queues
igc5: Using MSI-X interrupts with 5 vectors
igc5: Ethernet address: 00:90:27:e8:33:24
igc5: netmap queues/slots: TX 4/1024, RX 4/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
hdac0: <Intel Tiger Lake HDA Controller> mem 0x6001128000-0x600112bfff,0x6001000000-0x60010fffff irq 16 at device 31.3 on pci0
pci0: <serial bus> at device 31.5 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> on acpi0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
hwpstate_intel4: <Intel Speed Shift> on cpu4
hwpstate_intel5: <Intel Speed Shift> on cpu5
hwpstate_intel6: <Intel Speed Shift> on cpu6
hwpstate_intel7: <Intel Speed Shift> on cpu7
Timecounter "TSC-low" frequency 1401614042 Hz quality 1000
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
I tracked this majority of the extra CPU usage to the new gateway_watcher.php script on my firewall. I made a temporary fix by editing that script (it is at "/usr/local/opnsense/scripts/routes/gateway_watcher.php") and changed the line near the beginning from "$poll = 1; /* live poll interval */" to "$poll = 5; /* live poll interval */" to go from 1 second polling to 5 second polling and it dropped the usage down to around 4% 5 minute average from the 12% average I was seeing at the default polling.
It looks like that script forces a refresh of the configuration and parses it every second by default and if you have a large configuration it can impact the CPU significantly.
As promised I improved the configuration reload by offloading it to SIGHUP here:
https://github.com/opnsense/core/commit/b94097567cb
This WILL NOT apply using opnsense-patch since the PHP module pcntl is currently not in our build so it'll be added in 23.7.3.
Cheers,
Franco
thanks franco much appreciated!
Looks like it can be tested already
fetch https://pkg.opnsense.org/FreeBSD:13:amd64/snapshots/latest/All/php82-pcntl-8.2.9.pkg
pkg install php82-pcntl-8.2.9.pkg
opnsense-patch b94097567cb
or as a one liner
fetch https://pkg.opnsense.org/FreeBSD:13:amd64/snapshots/latest/All/php82-pcntl-8.2.9.pkg && pkg install php82-pcntl-8.2.9.pkg && opnsense-patch b94097567cbFull output here
Quoteroot@OPNsense:~ # fetch https://pkg.opnsense.org/FreeBSD:13:amd64/snapshots/latest/All/php82-pcntl-8.2.9.pkg
php82-pcntl-8.2.9.pkg 18 kB 435 kBps 00s
root@OPNsense:~ # pkg install php82-pcntl-8.2.9.pkg
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating mimugmail repository catalogue...
mimugmail repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
php82-pcntl: 8.2.9 [unknown-repository]
Number of packages to be installed: 1
Proceed with this action? [y/N]: y
[1/1] Installing php82-pcntl-8.2.9...
Extracting php82-pcntl-8.2.9: 100%
=====
Message from php82-pcntl-8.2.9:
--
This file has been added to automatically load the installed extension:
/usr/local/etc/php/ext-20-pcntl.ini
root@OPNsense:~ # opnsense-patch b94097567cb
Fetched b94097567cb via https://github.com/opnsense/core
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From b94097567cbb116025f54772609eef7b9a8e3f4e Mon Sep 17 00:00:00 2001
|From: Franco Fichtner <franco@opnsense.org>
|Date: Thu, 24 Aug 2023 09:58:22 +0200
|Subject: [PATCH] system: defer config reload to SIGHUP in gateway watcher
|
|This should considerably lower CPU usage as reported a few times.
|We do need to bring in pcntl PHP module in order to get that done
|easily in the script.
|
|PR: https://forum.opnsense.org/index.php?topic=35219.0
|---
| Makefile | 3 ++-
| src/etc/inc/plugins.inc.d/dpinger.inc | 17 +++++++++++------
| src/opnsense/scripts/routes/gateway_watcher.php | 17 +++++++++++++----
| 3 files changed, 26 insertions(+), 11 deletions(-)
|
|diff --git a/src/etc/inc/plugins.inc.d/dpinger.inc b/src/etc/inc/plugins.inc.d/dpinger.inc
|index 048bd8a522..f12d5c06d9 100644
|--- a/src/etc/inc/plugins.inc.d/dpinger.inc
|+++ b/src/etc/inc/plugins.inc.d/dpinger.inc
--------------------------
Patching file etc/inc/plugins.inc.d/dpinger.inc using Plan A...
Hunk #1 succeeded at 319.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/src/opnsense/scripts/routes/gateway_watcher.php b/src/opnsense/scripts/routes/gateway_watcher.php
|index e67debbff2..248650205b 100755
|--- a/src/opnsense/scripts/routes/gateway_watcher.php
|+++ b/src/opnsense/scripts/routes/gateway_watcher.php
--------------------------
Patching file opnsense/scripts/routes/gateway_watcher.php using Plan A...
Hunk #1 succeeded at 31.
Hunk #2 succeeded at 54.
done
All patches have been applied successfully. Have a nice day.
Yep, uploaded the new snapshot build today. Still need to restart the watcher:
# pluginctl -s dpinger restart :watcher:
Cheers,
Franco
In my case the patch did not help to reduce CPU usage at all. The change of the $poll from 1 to 5 helped to get CPU usage back to the level it was with 23.1.
It really doesn't do much as I said initially. It also depends on hardware... or when a link is flapping constantly and it's forced to trigger. But nobody posted a gateway log of such events. These are new in 23.7 by the way to see exactly what happened to a gateway and when.
We can make the delays configurable some day but not before ironing out the real issues first.
Cheers,
Franco
Sorry, I did somehow miss your request for entries out of the log file. I am not sure if this helps, it is out of /var/log/gateways/gateways_2023082{4,6}.log (the newest 2 files):
<165>1 2023-08-24T13:18:04+00:00 scrambler.wenks.ch dpinger 2426 - [meta sequenceId="1"] ALERT: WAN1IP4GW (Addr: 8.8.8.8 Alarm: down -> none RTT: 2.7 ms RTTd: 1.3 ms Loss: 0.0 %)
<165>1 2023-08-24T13:18:04+00:00 scrambler.wenks.ch dpinger 2426 - [meta sequenceId="2"] ALERT: WAN2IP4GW (Addr: 8.8.4.4 Alarm: down -> none RTT: 15.1 ms RTTd: 6.8 ms Loss: 0.0 %)
<165>1 2023-08-24T13:18:04+00:00 scrambler.wenks.ch dpinger 2426 - [meta sequenceId="3"] ALERT: WAN1IP6GW (Addr: 2606:4700:4700::1111 Alarm: down -> none RTT: 2.4 ms RTTd: 1.5 ms Loss: 0.0 %)
<165>1 2023-08-24T13:18:04+00:00 scrambler.wenks.ch dpinger 2426 - [meta sequenceId="4"] ALERT: WAN2IP6GW (Addr: 2620:fe::9 Alarm: down -> none RTT: 2.5 ms RTTd: 1.5 ms Loss: 0.0 %)
<165>1 2023-08-24T18:18:52+00:00 scrambler.wenks.ch dpinger 72524 - [meta sequenceId="1"] ALERT: WAN1IP4GW (Addr: 8.8.8.8 Alarm: down -> none RTT: 2.3 ms RTTd: 1.3 ms Loss: 0.0 %)
<165>1 2023-08-24T18:18:52+00:00 scrambler.wenks.ch dpinger 72524 - [meta sequenceId="2"] ALERT: WAN2IP4GW (Addr: 8.8.4.4 Alarm: down -> none RTT: 12.5 ms RTTd: 4.9 ms Loss: 0.0 %)
<165>1 2023-08-24T18:18:52+00:00 scrambler.wenks.ch dpinger 72524 - [meta sequenceId="3"] ALERT: WAN1IP6GW (Addr: 2606:4700:4700::1111 Alarm: down -> none RTT: 2.1 ms RTTd: 1.5 ms Loss: 0.0 %)
<165>1 2023-08-24T18:18:52+00:00 scrambler.wenks.ch dpinger 72524 - [meta sequenceId="4"] ALERT: WAN2IP6GW (Addr: 2620:fe::9 Alarm: down -> none RTT: 2.1 ms RTTd: 1.3 ms Loss: 0.0 %)
<12>1 2023-08-26T15:24:50+00:00 scrambler.wenks.ch dpinger 50861 - [meta sequenceId="1"] exiting on signal 15
<12>1 2023-08-26T15:24:50+00:00 scrambler.wenks.ch dpinger 60121 - [meta sequenceId="2"] send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.8.8 bind_addr 62.12.173.5 identifier "WAN1IP4GW "
<165>1 2023-08-26T15:24:51+00:00 scrambler.wenks.ch dpinger 72524 - [meta sequenceId="3"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2023-08-26T15:25:05+00:00 scrambler.wenks.ch dpinger 57225 - [meta sequenceId="4"] exiting on signal 15
<12>1 2023-08-26T15:25:06+00:00 scrambler.wenks.ch dpinger 82049 - [meta sequenceId="5"] send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 8.8.4.4 bind_addr 62.2.85.182 identifier "WAN2IP4GW "
<165>1 2023-08-26T15:25:08+00:00 scrambler.wenks.ch dpinger 72524 - [meta sequenceId="6"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2023-08-26T15:26:26+00:00 scrambler.wenks.ch dpinger 48560 - [meta sequenceId="1"] exiting on signal 15
<12>1 2023-08-26T15:26:26+00:00 scrambler.wenks.ch dpinger 68631 - [meta sequenceId="2"] send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 2606:4700:4700::1111 bind_addr 2001:8a8:1005:1::5 identifier "WAN1IP6GW "
<165>1 2023-08-26T15:26:26+00:00 scrambler.wenks.ch dpinger 72524 - [meta sequenceId="3"] Reloaded gateway watcher configuration on SIGHUP
<12>1 2023-08-26T15:26:39+00:00 scrambler.wenks.ch dpinger 52964 - [meta sequenceId="4"] exiting on signal 15
<12>1 2023-08-26T15:26:40+00:00 scrambler.wenks.ch dpinger 59088 - [meta sequenceId="5"] send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% alarm_hold 10000ms dest_addr 2620:fe::9 bind_addr 2001:8a8:1005:2::182 identifier "WAN2IP6GW "
<165>1 2023-08-26T15:26:40+00:00 scrambler.wenks.ch dpinger 72524 - [meta sequenceId="6"] Reloaded gateway watcher configuration on SIGHUP
<165>1 2023-08-26T16:16:36+00:00 scrambler.wenks.ch dpinger 56541 - [meta sequenceId="1"] ALERT: WAN1IP4GW (Addr: 8.8.8.8 Alarm: down -> none RTT: 2.3 ms RTTd: 2.0 ms Loss: 0.0 %)
<165>1 2023-08-26T16:16:36+00:00 scrambler.wenks.ch dpinger 56541 - [meta sequenceId="2"] ALERT: WAN2IP4GW (Addr: 8.8.4.4 Alarm: down -> none RTT: 13.4 ms RTTd: 6.5 ms Loss: 0.0 %)
<165>1 2023-08-26T16:16:36+00:00 scrambler.wenks.ch dpinger 56541 - [meta sequenceId="3"] ALERT: WAN1IP6GW (Addr: 2606:4700:4700::1111 Alarm: down -> none RTT: 1.9 ms RTTd: 2.1 ms Loss: 0.0 %)
<165>1 2023-08-26T16:16:36+00:00 scrambler.wenks.ch dpinger 56541 - [meta sequenceId="4"] ALERT: WAN2IP6GW (Addr: 2620:fe::9 Alarm: down -> none RTT: 2.1 ms RTTd: 2.1 ms Loss: 0.0 %)
That looks normal. Thanks so far.
I've slipped a small change into 23.7.3 to be able to optimise this further. If you are on 23.7.3 you can try the next patch:
https://github.com/opnsense/core/commit/bb951f2eb
# opnsense-patch bb951f2eb
# pluginctl -s dpinger restart :watcher:
Note that this breaks if being applied to 23.7.2 or earlier...
Cheers,
Franco
It helped to reduce CPU usage. In my case it is around 1/3 down from the 23.7.0 level to what it was with 23.1.x level (both with default $poll = 1).
That's sort of what I saw from here... for each second I had 0.3 seconds overall processing time and that patch should take 0.1 away. I still have to look at the class - that seems to be where the other 0.2 seconds are spent. Maybe we can get to down another 0.1. Otherwise we really have to change the standard poll interval.
Cheers,
Franco
Hello Franco
On my systems since the update to OPNsense 23.7.4 last night the CPU usage is down to the level it was with 23.1.x.
Thank you very much for this improvements!
Best regards,
Fabian
I am seeing high CPU use and temperatures after upgrading to 23.7.9. Netflow seems to be the culprit. Resetting Netflow data and rebuilding the database have not helped, nor has rebooting the box.
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
15527 root 1 100 0 30M 20M CPU3 3 18:40 93.46% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.9)
88671 root 31 20 0 865M 203M uwait 2 1:13 3.42% /usr/local/AdGuardHome/AdGuardHome -s run
26850 root 1 52 0 58M 28M accept 2 0:02 1.76% /usr/local/bin/php-cgi
56115 root 1 20 0 13M 2548K bpf 3 0:09 1.27% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
32301 root 5 20 0 45M 12M kqread 0 0:19 0.88% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
18061 root 1 52 0 59M 28M accept 0 0:06 0.59% /usr/local/bin/php-cgi
23558 root 1 20 0 59M 28M accept 2 0:04 0.49% /usr/local/bin/php-cgi
Has anyone else seen this?
After a couple of days' running, it still seems netflow is causing significantly higher CPU use than before the upgrade. I have turned off IPv6 completely (following these (https://www.thomas-krenn.com/en/wiki/OPNsense_disable_IPv6) instructions) to see if that would help; it made no difference.
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
15527 root 1 80 0 54M 36M CPU2 2 22.8H 27.78% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.9)
51856 root 1 21 0 58M 28M accept 3 0:05 1.27% /usr/local/bin/php-cgi
66423 root 1 52 0 58M 28M accept 3 0:02 1.17% /usr/local/bin/php-cgi
83323 root 1 20 0 58M 28M accept 2 0:05 0.29% /usr/local/bin/php-cgi
88671 root 31 20 0 1130M 477M uwait 0 38:39 0.10% /usr/local/AdGuardHome/AdGuardHome -s run
18068 root 1 52 0 58M 29M accept 1 0:06 0.10% /usr/local/bin/php-cgi
14297 root 1 25 0 13M 3792K pause 3 0:00 0.10% /bin/csh
241 root 7 52 0 104M 42M accept 2 19:32 0.00% /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.9)
20283 root 1 20 0 88M 60M nanslp 1 11:59 0.00% /usr/local/bin/php /usr/local/opnsense/scripts/routes/gateway_watcher.php interface routes alarm
32301 root 4 20 0 49M 12M kqread 3 8:51 0.00% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
73875 root 1 20 0 13M 2644K bpf 3 4:38 0.00% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
79273 root 1 25 0 13M 3016K wait 3 3:14 0.00% /bin/sh /var/db/rrd/updaterrd.sh
Does anyone have suggestions as to what I could attempt, other than turning netflow off?
Same issue here. Since upgrade to 23.7.10_1-amd64 cpu usage is much more higher then before and my monit system monitor regularly complains about cpu usage over 75%.
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
46114 root 1 85 0 48M 31M CPU1 1 33.0H 97.91% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.9)
24479 root 1 28 0 69M 33M nanslp 0 0:01 11.18% /usr/local/bin/php-cgi
257 root 1 52 0 135M 40M accept 0 39:54 4.87% /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.9)
67545 root 13 20 0 2869M 1074M nanslp 1 790:15 3.19% /usr/local/bin/suricata -D --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
19874 root 12 20 -20 906M 43M uwait 2 124:11 2.45% ipdrstreamer
22851 mongodb 28 52 0 2876M 1043M uwait 2 242:15 2.34% /usr/local/bin/mongod --logappend --setParameter=disabledSecureAllocatorDomains=* --config /usr/local/etc/mongodb.conf --fork
19514 root 12 20 -20 2997M 332M nanslp 1 471:32 2.03% eastpect: Eastpect Instance 0 (eastpect)
55207 root 3 20 0 50M 9976K kqread 3 12:06 0.46% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
15107 root 1 20 0 14M 3660K CPU3 3 0:02 0.43% top
44068 root 1 20 0 60M 31M accept 0 0:06 0.31% /usr/local/bin/php-cgi
37823 root 2 20 -20 946M 56M nanslp 2 18:10 0.28% /usr/local/zenarmor//bin/eastpect -D
80154 root 1 20 0 13M 1796K bpf 0 2:04 0.18% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
80838 root 1 28 0 13M 1784K nanslp 0 0:19 0.10% /usr/sbin/cron -s
16942 root 1 20 0 21M 7204K kqread 1 1:04 0.09% /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
69045 root 1 20 0 25M 14M select 1 1:58 0.08% /usr/local/bin/python3 /usr/local/opnsense/scripts/dhcp/unbound_watcher.py --domain home (python3.9)
82347 root 1 20 0 23M 11M select 1 3:23 0.05% /usr/local/bin/python3 /usr/local/sbin/configctl -e -t 0.5 system event config_changed (python3.9)
98727 root 1 20 0 18M 7344K select 0 0:00 0.05% sshd: root@pts/0 (sshd)
82850 root 1 20 0 23M 11M select 1 3:04 0.04% /usr/local/bin/python3 /usr/local/opnsense/scripts/syslog/lockout_handler (python3.9)
258 squid 1 20 0 174M 21M kqread 3 1:39 0.03% (squid-1) --kid squid-1 -f /usr/local/etc/squid/squid.conf (squid)
54088 root 1 21 0 13M 1932K wait 1 1:25 0.02% /bin/sh /var/db/rrd/updaterrd.sh
39938 root 1 20 0 21M 5868K select 2 0:28 0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
Does anyone have some suggestions?
Greetings Manuel