OPNsense Forum
Archive => 20.7 Legacy Series => Topic started by: lore.phoenix on August 17, 2020, 02:32:51 pm
-
Hello,
My OPNSense system has the CPU type: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2 cores)
Before I upgraded to OPNSense version 20.7 I was able to see the temperature at the Thermal Sensors widget on my OPNSense (v20.1.9) dashboard.
After the upgrade I waited serveral hours but the Therminal Sensors widget on my OPNSense (v20.7.1) dashboard doesn't display anything.
I checked under Reporting -> Health -> System -> Cputemp and the Cputemp graph is able to display the temperature.
The actions I already took :
1. After that I set the Thermal Sensor under System -> Settings -> Miscellaneous to None/ACPI.
I rebooted the server.
2. I enabled the Thermal Sensors under System -> Settings -> Miscellaneous to
Hardware : Intel Core* CPU on-die therminal sensor (coretemp).
On the CLI I saw the following message appearing:
# coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 92a092a0600092a
device_attach: est1 attach returned 6
# sysctl -a | grep temperature
dev.cpu.1.temperature: 35.0C
dev.cpu.0.temperature: 32.0C
3. After 15 ~ 20mins I was able to see new Cputemp graphs under Reporting -> Health -> System -> Cputemp but the Thermal Sensor widget still doesn't display anything.
4. I removed the Thermal Sensor widget from my dashboard and re-add the Thermal Sensor widget on my dashboard but the result is the same (meaning no data).
Since I am able to read the temperature using the CLI command 'sysctl -a | grep temperature' and able to get the Cputemp graph under Reporting -> Health -> System -> Cputemp that I assume that the problem is related on the Thermal Sensors widget.
Is it possible to fix this ??
Kind regards,
-
Hi everyone,
Same issue as lore.phoenix (not showing anything on the widget), except that on Reporting -> Health -> System -> Cputemp, it does show only one graph from cpu0temp, whereas my router have 4 cores
sysctl -a | grep -E "dev.*temperature|hw.*temperature"
hw.acpi.thermal.tz1.temperature: 29.9C
hw.acpi.thermal.tz0.temperature: 27.9C
dev.cpu.3.temperature: 51.0C
dev.cpu.2.temperature: 51.0C
dev.cpu.1.temperature: 50.0C
dev.cpu.0.temperature: 50.0C
but not such error on system.log as shown by lore.phoenix
I am new to freeBSD and opnsense (i did not do any update, it an install from scratch), i will try to search for any error but any help would be great :)
have a good day
-
Click the pencil on the temp widget, there's a check box in the widget settings for all or single core.
-
Just for what it is worth, I stumbled across this. I just updated last night. I'm still seeing the temps. I'm using: Intel(R) Celeron(R) CPU J3160 @ 1.60GHz (4 cores)
-
Click the pencil on the temp widget, there's a check box in the widget settings for all or single core.
Many thanks Marjohn56! I have done a factory reset of the bios settings + re-activating core temp + done what you suggested, and it did work well ! :)
For reference, I have a mini pc Qotom with 6 LAN ports and i3-6100u cpu
thanks again for your quick help and have a nice day
-
Quote from: marjohn56 on August 21, 2020, 11:20:15 pm
Click the pencil on the temp widget, there's a check box in the widget settings for all or single core.
I tried what marjohn56 suggested but that didn't work for me.
It was disabled but even enabling the option 'Only show first found CPU core temperature' didn't display anything.
Quote from: chukkyuggets on August 22, 2020, 10:02:40 pm
... have done a factory reset of the bios settings + re-activating core temp ...
@chukkyuggets
After my BIOS reset + reactivating core temp that it fixed my issue. Many thanks for your input/solution.
My hardware:
Motherboard: Gigabyte P35-DS3R
Processor: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2 cores)
-
A little confused. Why is a BIOS reset required to get the thermal data showing again after the 20.7 upgrade?
I'm not even sure how I perform a factory reset on my box (PCEngine APU2). ???
-
Same here. Running on a ProtectLI FW6 with an i3 CPU.
Report --> Health --> CPU Temp is showing information, but the "Thermal Sensors" widget on the dashboard is empty.
I have the Thermal Sensor hardware set to coretemp, and it seems to be working per the shell command:
root@OPNsense:~ # sysctl -a | grep -E "dev.*temperature|hw.*temperature"
hw.acpi.thermal.tz1.temperature: 29.9C
hw.acpi.thermal.tz0.temperature: 27.9C
dev.cpu.3.temperature: 52.0C
dev.cpu.2.temperature: 52.0C
dev.cpu.1.temperature: 51.0C
dev.cpu.0.temperature: 51.0C
EDIT: It's ridiculous to have to do a BIOS reset and then reconfigure BIOS settings again when the system is clearly reading the temps (see right above). There is something broken in the OPNsense GUI. There has to be a better solution.
-
Word of note. 20.7 has moved to FreeBSD 12.1, there may be something there that is affecting some devices. I'm running Qotom's and have had no issues.
-
@marjohn56
20.7 has moved to HardenBSD 12.1 which is FreeBSD with more security in mind.
I would be surprised that the problem is related on the OS since I was able to read the temperature via CLI at the moment the Thermal Sensor widget didn't display anything.
I would like to know how the Thermal Sensor widget read the temperature.
Does it read the temperature directly from the system or does it read it's data from the stored database that is used for Report --> Health --> CPU Temp.
Kind regards,
-
> I would like to know how the Thermal Sensor widget read the temperature.
# /sbin/sysctl -a | grep temperature
Cheers,
Franco
-
I'm seeing a lot of lines like the first then the two at the end..
# /sbin/sysctl -a | grep temperature
<118>2020-08-14T19:01:59.592981+01:00 OPNsense.my.domain.uk lighttpd 56725 - - xx.xxx.xxx.xx xx.xxx.xxx.x - [14/Aug/2020:19:01:59 +0100] "GET /widgets/api/get.php?load=system%2Cgateway%2Cinterfaces%2Ctemperature&_=1597428113024 HTTP/1.1" 200 3254 "https://xx.xxx.xxx.xx/index.php" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
..
..
dev.cpu.1.temperature: 52.2C
dev.cpu.0.temperature: 52.2Con an HP T610, AMD G-T56N
-
Odd seems to be picking up web stuff.
try
# /sbin/sysctl -a | grep temperature:
instead of
# /sbin/sysctl -a | grep temperature
*** EDIT***
What's even more odd is sysctl outputting anything that has web stuff in it.
-
try
# /sbin/sysctl -a | grep temperature:
# /sbin/sysctl -a | grep temperature:
dev.cpu.3.temperature: 53.2C
dev.cpu.2.temperature: 53.2C
dev.cpu.1.temperature: 53.2C
dev.cpu.0.temperature: 53.2C
-
Yes, so the grep is seeing the 'temperature' in the junk the sysctl appears to ouput. Can you do this, only if you are having problems with CPU temp:
# /sbin/sysctl -a > /tmp/sysctrl_output.txt and then post the resultant text file here. It'll be interesting to see what's going on. Also give make model of hardware and amount of ram etc.
-
Yes, so the grep is seeing the 'temperature' in the junk the sysctl appears to ouput. Can you do this, only if you are having problems with CPU temp:
# /sbin/sysctl -a > /tmp/sysctrl_output.txt and then post the resultant text file here. It'll be interesting to see what's going on. Also give make model of hardware and amount of ram etc.
sysctrl output attached. Have also attached response from /widgets/api/get.php?load=temperature (not sure if that's helpful at all).
System is: PC Engines APU2 AMD GX-412TC SOC 4GB
-
Assume that was for me?
I have graphs but no temps in web GUI. All 2.1 versions I had GUI and graphs
Edit: HP T610, 4096 MB, 16GB SATA Flash Drive
-
@Fruit - Your text file is full of all sorts of junk, half of the system log file is in there too! Something nasty going on there, surprised the system works at all.
-
@nigelw - Yours looks correct, but you are not seeing any temperatures in the widget?
-
@nigelw - Yours looks correct, but you are not seeing any temperatures in the widget?
Widget just shows the header. :(
-
Going to fire up my old APU2, not had a firmware update for years... let's see what happens.
-
Oh. Don't know why I didn't think of it earlier, but decided to turn the console (browser) on. There's this:
Error: Syntax error, unrecognized expression: #thermal_sensors_widget_<118>2020-08-18T22
at Function.se.error (jquery-3.5.1.min.js:2)
at se.tokenize (jquery-3.5.1.min.js:2)
at se.select (jquery-3.5.1.min.js:2)
at Function.se [as find] (jquery-3.5.1.min.js:2)
at S.fn.init.find (jquery-3.5.1.min.js:2)
at <anonymous>:19:23
at Array.map (<anonymous>)
at thermal_sensors_widget_update (<anonymous>:6:14)
at index.php:1351
at Array.map (<anonymous>)
at Object.<anonymous> (index.php:1348)
at c (jquery-3.5.1.min.js:2)
at Object.fireWith [as resolveWith] (jquery-3.5.1.min.js:2)
at l (jquery-3.5.1.min.js:2)
at XMLHttpRequest.<anonymous> (jquery-3.5.1.min.js:2)
-
OK, what browser is that?
-
@Fruit - Your text file is full of all sorts of junk, half of the system log file is in there too! Something nasty going on there, surprised the system works at all.
Oh dear, I'll try a reboot. I'm in the middle of moving house so it might have to struggle on for a bit. Seems to be functioning OK
-
Have you done the syslog-ng patcch? Do you see syslog failing?
-
There's also syslog garbage in the HTML output...
Error: Syntax error, unrecognized expression: #thermal_sensors_widget_<118>2020-08-18T22
-
There's also syslog garbage in the HTML output...
Error: Syntax error, unrecognized expression: #thermal_sensors_widget_<118>2020-08-18T22
yes, fun isn't it. :)
memory or hd would be my first port of call.
-
@nigelw - still going through the upgrade process!
It's a bit slooooow!
-
Maybe syslog-ng isn't running at all so the messages try to find a way "home". ;)
-
@nigelw - Updated from 19.7 something all the way to 20.7.1, here's my lobby image.
(https://i.ibb.co/2jrS5sf/Capture.png) (https://ibb.co/XzPDV3v)
-
Maybe syslog-ng isn't running at all so the messages try to find a way "home". ;)
Hence my first thought about syslog-ng, but more of a corruption event happening at the same time.
-
Yeah, nice catch actually. That would explain why temperature reading is fine. I checked earlier today.
Something else interfering with the reads, probably causing numerous subtle unrelated issues on the side.
Cheers,
Franco
-
I'll leave my APU running on test for a while and see if it plays up. No patches, just a straight update to 20.7.1, if it is syslog-ng then it should show up soon enough.
-
Sorry, @marjohn56. Thanks for testing. I had to go to bed eventually. I've picked it up again and taken a closer look at things. Here's my analysis:
The sysctrl_output.txt I attached previously may not have been entirely helpful. Not sure if @fruit's would have been more helpful. I didn't check. Anyhoo...
The sysctrl dump contained a value for kern.msgbuf with a whole lot of junk, including what seems to be web request logs. I removed them because they contained a bunch of IPs and domain names which I didn't want to go through and sanitise. Those logs include requests (presumably from the dashboard) to /widgets/api/get.php?load=system%2Ctraffic%2Ctemperature%2Cgateway%2Cinterfaces...
<118>2020-08-18T22:38:44.491556+10:00 firewall.xxxx lighttpd 79007 - - xxxx firewall.xxxx - [18/Aug/2020:22:38:44 +1000] "GET /widgets/api/get.php?load=system%2Ctraffic%2Ctemperature%2Cgateway%2Cinterfaces&_=1597753466133 HTTP/1.1" 200 8670 "http://firewall.xxxx/index.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36"
<118>2020-08-18T22:38:51.924012+10:00 firewall.xxxx lighttpd 79007 - - xxxx firewall.xxxx - [18/Aug/2020:22:38:51 +1000] "GET /widgets/api/get.php?load=system%2Ctraffic%2Ctemperature%2Cgateway%2Cinterfaces&_=1597753466134 HTTP/1.1" 200 8671 "http://firewall.xxxx/index.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36"
<118>2020-08-18T22:38:59.269693+10:00 firewall.xxxx lighttpd 79007 - - xxxx firewall.xxxx - [18/Aug/2020:22:38:59 +1000] "GET /widgets/api/get.php?load=system%2Ctraffic%2Ctemperature%2Cgateway%2Cinterfaces&_=1597753466135 HTTP/1.1" 200 8670 "http://firewall.xxxx/index.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36"
When the temperature widget API does a grep for "temperature", it's picking up all these lines as well as the correct temperature values. When the widget tries to update the progress bars, it starts looking for elements with weird names and JQuery spits the dummy. Hence, this error:
Syntax error, unrecognized expression: #thermal_sensors_widget_<118>2020-08-18T22
I have no idea why kern.msgbuf is there or if/why it's different to what it used to be. But that's definitely the issue (for me, at least). I would suggest tightening the grep expression to exclude the log lines and just return the values we're actually looking for.
I've never created a PR for an open source project before, so I thought I'd give it a crack (even for a single character change ;)).
https://github.com/opnsense/core/pull/4300
-
Nigelw,
Nice catch. I also have a "kern.msgbuf" section with a handful of lines like these in it (note the "temperature" string embedded in there):
<118>2020-08-26T01:47:36.967822-07:00 OPNsense.kurort lighttpd 61178 - - 172.16.1.106 172.16.1.1 - [26/Aug/2020:01:47:36 -0700] "GET /widgets/api/get.php?load=traffic%2Cinterfaces%2Cgateway%2Csystem%2Ctemperature&_=1598431650218 HTTP/1.1" 200 11921 "https://172.16.1.1/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0"
This is probably causing the widget to fail, even while this works:
root@OPNsense:/tmp # /sbin/sysctl -a | grep temperature:
hw.acpi.thermal.tz1.temperature: 29.9C
hw.acpi.thermal.tz0.temperature: 27.9C
dev.cpu.3.temperature: 53.0C
dev.cpu.2.temperature: 53.0C
dev.cpu.1.temperature: 52.0C
dev.cpu.0.temperature: 52.0Croot@OPNsense:/tmp #
-
I hadn't intended posting again this soon but though this might help in some way - of course it may not help at all but I may not have much time to get back here for a while
I did a couple of reboots last night (an extra one just for good measure ;) and all is looking good, I have temps in GUI and graphs and
/sbin/sysctl -a | grep temperature
dev.cpu.1.temperature: 55.1C
dev.cpu.0.temperature: 55.1Cnone of those extraneous lines
/sbin/sysctl -a is also cleaner but still some lighttpd lines
I am still concerned that I may have memory/disk issues so will be keeping a close eye on things
-
@fruit - Thanks for update. We are now aware that there appears to be extraneous data being returned when sysctl -a is called, this is the route of the problem and we're trying to find out why. Why it's not happening on every system is also a bit of a mystery
-
running:
sysctl kern.msgbuf_clear=1... on the shell/console clears it, and the temperatures INSTANTLY start showing up on the dashboard.
-
Thanks Xelas, question is, why is there junk in there to start with. Might be a 12.1 issue that needs to be resolved, but that call before any other calls using sysctl would be a good idea
-
From what I can tell, the last few KB from dmesg get saved in there whenever there is a system crash to aid in debugging, and this has been the behavior of FreeBSD since at least 9.0 Why this has suddenly become an issue now is mystery. Perhaps we were lucky and nothing containing the text "temperature" made it in there before to break the widget. Who knows.
I found this article from 2011 explaining how to clear kern.msgbuf:
https://www.gnutoolbox.com/clearing-dmesg-logs/
-
This makes it more likely that the problem is/was with syslog-ng. Franco has patched this and it's in for the 20.7.2 release, however the patch is available now, so it could be a good idea to run this:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/syslog-ng327-3.27.1_2.txz
-
This makes it more likely that the problem is/was with syslog-ng. Franco has patched this and it's in for the 20.7.2 release, however the patch is available now, so it could be a good idea to run this:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/syslog-ng327-3.27.1_2.txz
In case it helps others, please can someone confirm that from 20.7.1 the changes suggested in the console after
syslog-ng is now installed! To replace FreeBSD's standard syslogd
(/usr/sbin/syslogd), complete these steps:are not required? I tried them last night, something broke and it froze so I reverted but left syslog-ng327-3.27.1_2.txz in then a reboot and all has been well since.
-
Ignore that waffle.. :)
A few hours have been spent by myself and @nigelw looking at better methods of parsing the temperature info from sysctl, we have a few runners so that will also be happening.
-
Thanks Xelas, question is, why is there junk in there to start with. Might be a 12.1 issue that needs to be resolved, but that call before any other calls using sysctl would be a good idea
If you are parsing sysctl output, then you can never be sure that kern.msgbuf does not contain a string that won't trip your parser. IMHO, unless that info is truly useful for crash diagnosis, I'd clear it with a "sysctl kern.msgbuf_clear=1" before parsing it.
In any case, looks like you got it. I just wanted to get this thought out of my head. :-)
You guys are doing an amazing job staying on top of all of the issues and bugs. OPNsense is a very complicated project!
-
Yes, but I personally don't really want to kill that data, it could be useful for debug purposes. The solution is tighter sysctl requests and parsing of some form. There are several options in play, the dev's will make their choice and that should see the end of this issue once and for all.
-
Yes, but I personally don't really want to kill that data, it could be useful for debug purposes.
Agreed. A dashboard widget should not be responsible for removing unrelated data from the system. IMHO.
-
Just wanted to add for anyone else administering an OPNSense server that needs to see temperature sensors:
As seen prior in this thread: sysctl -a | grep temperature
IPMI Tool: ipmi-sensors
To install ipmi tool:
opnsense-code tools ports src
cd /usr/src
git checkout stable/21.7
pkg install autoconf automake libtool
cd /usr/ports/sysutils/freeipmi
make install clean
ipmi-sensors
Cheers everyone, would love to see more information about how to install racadm, hponcfg/hp-health, and imm for future deployments if you know anything about the setups for OPNSense baremetal installations with those tools, I'd love to see a post made in these forums.
Tmanok