Hi,
I have telemetry enabled for the ET Pro ruleset, but the telemetry widget fails to load with this error in the backend logs.
I'm running OPNsense in a VM for tesing and I am on the Dev branch.
Thanks for your help.
Can you run that command in single quotes manually and see what it complains about?
Cheers,
Franco
Sure, here's what it complains about (the screenshot is too large to be uploaded here) :
"root@ClosedSense:~ # /usr/local/opnsense/scripts/etpro_telemtry/sensor_info.py
Traceback (most recent call last):
File "/usr/local/opnsense/scripts/etpro_telemtry/sensor_info.py", line 33, in <module>
import telemetry
File "/usr/local/opnsense/scripts/etpro_telemtry/telemetry/__init__.py", line 34, in <module>
import netaddr
ModuleNotFoundError: No module named 'netaddr'
root@ClosedSense:~ # "
To my surprise you seem to be using the development version (and checking again you said so, sorry for not seeing it earlier) and it was already fixed for upcoming 25.7.4 but it also didn't break in 25.7.3:
https://github.com/opnsense/plugins/commit/877ebf20e
That's not patchable, but you can use the following for now:
# pkg install -A py311-netaddr
The -A is for automatic. If you perform an update it may be removed again but in 25.7.4 the commit above will cause the dependency to be registered again.
Cheers,
Franco
The widget for me never worked
https://forum.opnsense.org/index.php?topic=48493.msg244920#msg244920
yeah, it display "proofpoint" if you remove it then add it again but with no info and when you leave the dashboard and come back it fails to load again
Quote from: MaxMax99 on September 15, 2025, 10:46:12 PMyeah, it display "proofpoint" if you remove it then add it again but with no info and when you leave the dashboard and come back it fails to load again
Yep. It's broken.
Now a proper bug report would go a long way compared to bro-opting on an already fixed issue reported here walking back concluding "it" is "broken". I mean you just saw how the support process works and chose to undermine it.
Cheers,
Franco
Do you need any more infos guys?
Quote from: franco on September 16, 2025, 09:20:12 AMNow a proper bug report would go a long way compared to bro-opting on an already fixed issue reported here walking back concluding "it" is "broken". I mean you just saw how the support process works and chose to undermine it.
Cheers,
Franco
Again, a widget, which has nothing to do with the functionality of the actual plugin, hence I am not doing the effort to create a bug report for a non-working widget. I only do effort on functionality issues (i226-V nvm, text that is not readable in gui, etc), and even then I probably won't open a bug report.
I stated the facts, widget is(was) broken.
Where/when did it get fixed?
Cheerio.
> Where/when did it get fixed?
When did it get broken? Or more precisely, how? That's all I'm asking really.
Cheers,
Franco
Quote from: BrandyWine on September 16, 2025, 11:33:16 PMand even then I probably won't open a bug report
How do you expect things will get fixed if you don't open a bug report?
Quote from: Patrick M. Hausen on September 17, 2025, 09:05:09 AMQuote from: BrandyWine on September 16, 2025, 11:33:16 PMand even then I probably won't open a bug report
How do you expect things will get fixed if you don't open a bug report?
If I don't open a bug report it won't get fixed? That's a low probability, others will report it.
And, my fix was, delete widget from dashboard.
Quote from: franco on September 17, 2025, 09:01:48 AM> Where/when did it get fixed?
When did it get broken? Or more precisely, how? That's all I'm asking really.
Cheers,
Franco
I installed OPNsense at v25.1.12 (i think), widget didn't work there. Now I am at the latest community version.
I have to verify where I started by finding my initial download package.
I upgraded to 25.7.3_7 this morning, and I can confirm the Telemetry widget is now broken.
I don't know how to file a bug report, but I'm happy to do so if someone can point me at a How_To.
- login to github.com
- go to https://github.com/opnsense/plugins
- click on "Issues" (top menu bar)
- click on "New Issue" (top right green button)
- fill in the form
- send
OK, I created a bug report, first time doing that so I don't know if it's done properly but it describes the problem.
Thanks, just for reference the ticket is https://github.com/opnsense/plugins/issues/4943
The problem is most likely with emergingthreats. I reported it to their forum : https://community.emergingthreats.net/t/opnsense-suricata-rule-update-for-et-telemetry/1952/22
I don't think further mix and match doesn't work. It didn't work before when reporting bugs on top of bugs either.
Ok, my bad. The two are connected indeed, but the widget should show "failed" because the API poll also fails and is propagated, not show empty values (maybe it does on an older OPNsense version but again: details matter).
https://github.com/opnsense/plugins/issues/4943#issuecomment-3307339653
As of this morning, the widget does load, but it reports blank data. E.g. there is just a hyphen under each of the headings for Status, Last Event, etc.
I think it looks like they fixed the server config so can you redo the test in https://github.com/opnsense/plugins/issues/4943#issuecomment-3306314947 ?
Cheers,
Franco
I redid the test and got an OK result back this time. The widget does load, but displays a hyphen in each data field instead of the valid dates that seem to be getting passed back by the API.
Thinking it might help to restart the Intrusion Detection service, I did that and the firewall crashed, showing an endless loop of scrolling dump error messages.
I rebooted and everything is running OK at the moment. I did notice this in the boot messages "pid 31 (zpool) is attempting to use unsafe AIO requests - not logging anymore", and the dmesg.boot file ends on that message.
I did a bunch of searching on that message and the results vary, so at this point I'm going to do nothing, and at least I won't make it worse by pasting commands from the Internet into a working firewall. *nix is an endless series of rabbit holes. Always has been, since the early days when I was an HP-UX admin many decades ago.
UPDATE: The Suricata log is full of errors from the time of the crash. It shows that the engine restarted, and then within 3 seconds it started posting about 20,000 of these, until I shut it off.
[299151] -- igc0^: error reading netmap data via polling: No error: 0
[299176] -- igc0^: error reading netmap data via polling: No buffer space available
I guess there is a bit of confirmation bias at play here. We're debugging a widget, quickly pinpointed the server issue and Proofpoint fixed it. Maybe we're not there 100% yet but to throw in the towel seems like the most discouraging action here towards development and support effort.
Cheers,
Franco
I'm taking this to the next dev meeting:
https://github.com/opnsense/plugins/issues/4943#issuecomment-3312623289
Long story short is when the heartbeat couldn't be sent to the server due to server side SSL issue the status went "DORMAT" for users so the widget doesn't display anything by design. We can improve that design, sure, no question, but I think this also is much less important than it was set out to be.
Cheers,
Franco
OK, thank you.
Is there anything we as users can/should do to get the status from DORMANT back to ACTIVE?
According to the docs, we still receive the Pro rule set, even with DORMANT. https://docs.opnsense.org/manual/etpro_telemetry.html
Active status should be back with the next heartbeat acknowledged by Proofpoint.
We're in contact with them to make sure it's back to normal soon.
Cheers,
Franco
--
Quote from: franco on September 19, 2025, 05:09:11 PMI guess there is a bit of confirmation bias at play here. We're debugging a widget, quickly pinpointed the server issue and Proofpoint fixed it. Maybe we're not there 100% yet but to throw in the towel seems like the most discouraging action here towards development and support effort.
Cheers,
Franco
In this case, "doing nothing" means not making the matter worse by executing a bunch of "zpool restore -fingers -crossed" commands that I find on the Internet. I'll continue to monitor the firewall, watch this thread, and perform whatever commands the experts think might help.
As of this morning the widget data is populated again.
Hi,
The widget still dosen't load.
What about you guys?
Haven't updated the widget yet, but you can always inspect he API response if you need the information now.
Cheers,
Franco
Widget is still OK for me.
Here's the discussed change for 25.7.5:
https://github.com/opnsense/plugins/commit/a75a87d0b
That should show any type of status returned by the server.
Cheers,
Franco
Hi,
The widget now works proprely, thank you :)
In my suricata logs I see err 403's for heartbeat.py and other URI's for getting rules.
I am v25.7.3_7
Does the update to 25.7.5 fix all that (discussion has been about the widget). Am I right when I say the widget wasn't really the problem, but the plugin was. Is 25.7.5 a dependency for plugin v1.8 ?
These 403's keep showing up in system log file
<11>1 2025-10-09T03:12:49+00:00 Rice.localdomain send_heartbeat.py 39534 - [meta sequenceId="1"] unexpected result from https://opnsense.emergingthreats.net/api/v1/telemetry (http_code 403)
My widget still broken
(https://i.postimg.cc/cJNg0TKd/telem-broken.png)
(https://i.postimg.cc/TPXLfQyf/version.png)
(https://i.postimg.cc/25fbrTqm/plugin-version.png)
> Does the update to 25.7.5 fix all that (discussion has been about the widget).
No, the fix in 25.7.5 doesn't deal with invalid responses and I would consider a 403 a proper failed to load widget error in this case.
Cheers,
Franco
I currently removed the widget, and still see errors in log file
<11>1 2025-10-09T12:16:29+00:00 Rice.localdomain rule-updater.py 62960 - [meta sequenceId="116"] download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/version (http_code: 403)
Any call out to thst url (from the fw for suricata stuff) always gets 403.
403's are "not authorized" err, so maybe an issue with API key?
Could be, but wouldn't know why their server returned that.
Cheers,
Franco
Yes, maybe. Check API key.
Quote from: BrandyWine on October 09, 2025, 03:40:13 PMI currently removed the widget, and still see errors in log file
<11>1 2025-10-09T12:16:29+00:00 Rice.localdomain rule-updater.py 62960 - [meta sequenceId="116"] download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/version (http_code: 403)
Any call out to thst url (from the fw for suricata stuff) always gets 403.
403's are "not authorized" err, so maybe an issue with API key?
Quote from: MaxMax99 on October 15, 2025, 05:03:55 PMYes, maybe. Check API key.
With the 403's I suspect key issue, but it was working and now it's not, so if that's the case then that's odd, would seem my registered sensor was revoked.
From OPN github page (https://github.com/opnsense/docs/blob/master/source/manual/etpro_telemetry.rst) it's not clear to me how sensor registration is managed, but does seem clear the keys are generated & registered from OPNsense acct at proofpoint.