OPNsense Forum

Archive => 22.1 Legacy Series => Topic started by: Mundan101 on January 28, 2022, 02:10:36 pm

Title: Appliance running hotter after 22.1 update?
Post by: Mundan101 on January 28, 2022, 02:10:36 pm
I have a Qotom i7 that I've been running Opnsense on which consistently stayed about 50-55 prior with 21.7.  Now  its between 63-70 Celsius.

No errors, or anything out of the ordinary that I can tell so far.
Title: Re: Appliance running hotter after 22.1 update?
Post by: Gary7 on January 28, 2022, 05:48:40 pm
Using a shell, can you run "top" and see if you have a process that running at 100% ?

On my little APU2 after upgrade, unbound was running at 100% and never dropped to idle.

In Unbound DNS: Blocklist, I removed all of the blocklists and even disabled DNSBL.
When I applied the changes, unbound started and dropped to idle right away.
Then, I enabled DNSBL and added blocklists.
I add several blocklists, apply, and unbound starts, adds the blocklists, and drops to idle relatively quickly.
However, there is one very large (50 MB) blocklist that seems to cause constant 100% load for unbound. I'll investigate further this weekend.
Title: Re: Appliance running hotter after 22.1 update?
Post by: Mundan101 on January 29, 2022, 09:19:57 pm
Checked, nothing is running at 100%....

So far no performance lags in the network.
Title: Re: Appliance running hotter after 22.1 update?
Post by: ArminF on January 29, 2022, 09:49:44 pm
my quotom i7 -    Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz (4 cores, 8 threads)
start at 60-65 celsius and then goes down to 34.
Depending on CPU usage it stays between 30 and 50.

Yes i feel an increase as well.
But the throughout seems to be higher so i guessed it came from the suricata multi core update.
i do run suricata on wan. Lot of drop rules, DNSBL unbound and sensei on all the LAN interfaces.
Title: Re: Appliance running hotter after 22.1 update?
Post by: harshw on January 29, 2022, 10:37:40 pm
have a look at https://forum.opnsense.org/index.php?topic=26590.0. It could be that the CPU is running in turbo mode so runs hotter but will ramp down after a while. Or it could be that you need to enable SpeedShift or equivalent in the BIOS and make sure CPU Boot Performance is set to Max Non-Turbo Performance

If your dmesg shows hwpstate_intel as active, you can also tune it t achieve the temps you want via dev.hwpstate_intel.X.epp=[0-100] where 0 = best performance and 100=most energy efficient operation. Take a look at https://www.neelc.org/posts/freebsd-speed-shift-laptop/ and https://www.freebsd.org/cgi/man.cgi?query=hwpstate_intel&apropos=0&sektion=4&manpath=FreeBSD+13-current&arch=default&format=html
Title: Re: Appliance running hotter after 22.1 update?
Post by: Mundan101 on February 03, 2022, 12:48:46 am
So now upon a reboot i noticed that i received the core temp high warning.  The temps do go down after the system starts all services and becomes stable.



Title: Re: Appliance running hotter after 22.1 update?
Post by: allebone on February 05, 2022, 11:30:54 pm
I believe you are correct and replied to your post on reddit with my findings. Opnsense is slower if highadaptive is not used because under some cases it uses only 1 core and this causes issues with adaptive governor in powerd. This means you must run the appliance hotter to get performance in the test you described. There is no way around it if opnsense only uses 1 core for certain things.
Title: Re: Appliance running hotter after 22.1 update?
Post by: Mundan101 on February 08, 2022, 01:08:04 pm
Findings  show me the upgrade to 22.1 is more resource intensive, in the most simplistic newbie way i could determine.  Things that's worked under 21.7 is now pushing the CPU on 22.1 with my deployment hardware of choice.

I don't think any of my deployments are out of the ordinary however I have IDS/IPS, Zenarmor, Wireguard, OpnVPN, VLAN, telegraph, netdata and ntop running.  Majority of my deployments are Qotom i7-7500U and worked great!

I've tested a few boxes so far with 22.1 and it pushes the single core high in terms of usage.  1 box as already seen performance decrease, removing ntop helped stabilize.

Time for a hardware upgrade it seems....wahooo
Title: Re: Appliance running hotter after 22.1 update?
Post by: cephyr on February 09, 2022, 10:04:36 am
Hi,

I am experiencing the same thing, much more cpu usage than on 21 version.
I've opened top and it seems that ntopng was using much cpu. I disabled it now and see how it goes.


edit:

okay the biggset cpu impact is gone now but there is still a python process spiking cpu again and again

process in top:

24512 root          1  32    0    42M    29M select   3  73:13  36.43% python3.8



root@OPNsense:~ # procstat -f 24512
  PID COMM                FD T V FLAGS    REF  OFFSET PRO NAME
24512 python3.8         text v r r-------   -       - -   /usr/local/bin/python3.8
24512 python3.8          cwd v d r-------   -       - -   /
24512 python3.8         root v d r-------   -       - -   /
24512 python3.8            0 v c rw------   4       0 -   /dev/null
24512 python3.8            1 v c rw------   4       0 -   /dev/null
24512 python3.8            2 v c rw------   4       0 -   /dev/null
24512 python3.8            3 v r -w-----l   1       5 -   /var/run/flowd_aggregate.pid
24512 python3.8            4 v c rw------   4       0 -   /dev/null
24512 python3.8            5 s - rw------   1       0 UDD /var/run/logpriv


I am also running a Qotom i7 appliance
Title: Re: Appliance running hotter after 22.1 update?
Post by: karlson2k on February 10, 2022, 02:43:09 pm
Yes, I can see the similar.
Before 22.1 my CPU was always ~35-38 C with rare jumps up to 42 C.
After upgrade to 22.1, CPU stays at 48-52 C with periodic jumps to 39-41 C. I never saw temperature above 50 C before 22.1. Something definitely keeps freqs (and temperature) at much higher values than before.
Title: Re: Appliance running hotter after 22.1 update?
Post by: jepulismies on February 12, 2022, 03:35:00 pm
Can confirm this too, quite visible jump from average "idle" of 36'c to 40'c just by upgrading. Did an in-place upgrade with no (intentional) simultaneous config changes. Box is custom x86 PC with 4 GHz Intel CPU.

(https://i.imgur.com/AQ0hdvN.png)
Title: Re: Appliance running hotter after 22.1 update?
Post by: hescominsoon on February 12, 2022, 03:41:55 pm
I have noticed high cpu usage on my atom machine as well with the new version.  It's not pinning a core but i see constant spikes to 50 percent total usage.  It's usually python so i wonder why this newest version is doing in python that's so intensive.
Title: Re: Appliance running hotter after 22.1 update?
Post by: karlson2k on February 17, 2022, 05:03:33 pm
Obviously we have notable difference compared to previous version.

Any suggestion what to check?
Any terminal command?
Anything else?
Title: Re: Appliance running hotter after 22.1 update?
Post by: franco on February 18, 2022, 08:26:42 am
Has anyone benchmarked FreeBSD 12 vs. FreeBSD 13 yet? This could be a significant change, but nobody really asks why and how it impacts performance and system responsiveness.


Cheers,
Franco
Title: Re: Appliance running hotter after 22.1 update?
Post by: karlson2k on February 18, 2022, 09:14:36 am
There are several possible sources of temperature difference:
It could be any of these factors or combinations of them.

For me the throughput and the latency the same as before (the throughput is limited by my internet connection), no difference.

Maybe someone has spare set of equipment to make a clean OS test FreeBSD 12.1 / FreeBSD 13 (or even better: compare HardenedBSD 12.1 with FreeBSD 13). Unfortunately I don't have such ability.

However, I can make some other tests and checks on the live working equipment.
For me it looks like OPNsense 22.1 does not let system to be idle even without notable traffic, while with pervious versions of OPNsense system was almost idle with light traffic.

Could you advise me the best way how to check this assumption? Even negative result would be informative.
Title: Re: Appliance running hotter after 22.1 update?
Post by: franco on February 18, 2022, 09:20:02 am
3. The new version of OPNsense base framework may produce more CPU load
4. The new version of OPNsense may load CPU more

I wouldn't put my money on these two if we are already looking at "Python load" from a Python version that didn't even change its minor version between 21.7.8 and 22.1.


Cheers,
Franco
Title: Re: Appliance running hotter after 22.1 update?
Post by: karlson2k on February 18, 2022, 09:32:20 am
However, on other CPUs it may give different results. With switch from HardenedBSD to FreeBSD, OPNsense switched to the new compiler version and the new compiler settings therefore compiled binaries may change behavior on some CPUs.

Also possible that spikes from some process are kicking the CPU. Or maybe some short-living process fired regularly and keeps CPU hot.

I'm not a FreeBSD expert. Please advise how to exclude these assumptions.
Title: Re: Appliance running hotter after 22.1 update?
Post by: franco on February 18, 2022, 09:40:50 am
Well if by "The new version of OPNsense may load CPU more" you mean the "FreeBSD ports packages compiled by newer version of FreeBSD base compiler" I would agree with the labelling. Still, binary code doesn't significantly run hotter just because a new compiler is involved. It's most likely a FreeBSD kernel behavioural change.


Cheers,
Franco
Title: Re: Appliance running hotter after 22.1 update?
Post by: karlson2k on February 18, 2022, 06:09:06 pm
To summarize:
Can someone advise how to check and exclude (or confirm) numbers 4 and 5?
Title: Re: Appliance running hotter after 22.1 update?
Post by: franco on February 19, 2022, 08:51:31 am
Well if you don't take my word for it clone core.git from GitHub and...

# git checkout 22.1
# git diff 21.7.8


Happy hunting,
Franco
Title: Re: Appliance running hotter after 22.1 update?
Post by: allebone on February 21, 2022, 02:32:53 am
I would agree, its like installing the same app on windows 7, then upgrading to windows 8 and saying that the same app runs hotter/uses more cpu. Too much has changed under the hood to look at opnsense. The OS got bigger, uses more cpu cycles and with each new version of freebsd or whatever it will always tend to use more cpu as time goes on.
Title: Re: Appliance running hotter after 22.1 update?
Post by: Power Nabot on February 26, 2022, 10:25:00 pm
Hello,

First post here on this forum to say that, for me, it's the other way: 22.1 runs cooler than 21.7.
Started with 21.1, upgraded to 21.7, full reinstall for 22.1, and my temps went down from 58-61° to 52-53°.

Aliexpress appliance with 4 Intel i210 ports, Celeron J4125, 8GB DDR3, 128GB m-SATA SSD, running Zenarmor, 1 OpenVPN server, 1 OpenVPN client, Unbound blacklists and few FW rules. 20 devices max.
Title: Re: Appliance running hotter after 22.1 update?
Post by: Koldnitz on February 27, 2022, 06:23:13 pm
I have had the same experience as Power Nabot.

My temperatures went down across the board.  My load went up significantly.

I disabled PowerD since it did not seem to work as it did in 21.7 (CPU frequencies were idling very low)

Then I put this in tunables (from a link earlier in this thread):

Code: [Select]
dev.hwpstate_intel.0.epp=0
dev.hwpstate_intel.1.epp=0
dev.hwpstate_intel.2.epp=0
dev.hwpstate_intel.3.epp=0

This pegs my CPU (i7-7500U) to max frequency which reduces load.

Now my temps very slightly cooler and loading is similar or less to how it was in 21.7.

Cheers,