Telemetry widget fails to load on dashboard

[Patrick M. Hausen]

- login to github.com
- go to https://github.com/opnsense/plugins
- click on "Issues" (top menu bar)
- click on "New Issue" (top right green button)
- fill in the form
- send

[allenlook]

OK, I created a bug report, first time doing that so I don't know if it's done properly but it describes the problem.

[franco]

Thanks, just for reference the ticket is https://github.com/opnsense/plugins/issues/4943

[xavx]

The problem is most likely with emergingthreats. I reported it to their forum : https://community.emergingthreats.net/t/opnsense-suricata-rule-update-for-et-telemetry/1952/22

[franco]

I don't think further mix and match doesn't work. It didn't work before when reporting bugs on top of bugs either.

[franco]

Ok, my bad. The two are connected indeed, but the widget should show "failed" because the API poll also fails and is propagated, not show empty values (maybe it does on an older OPNsense version but again: details matter).

https://github.com/opnsense/plugins/issues/4943#issuecomment-3307339653

[allenlook]

As of this morning, the widget does load, but it reports blank data.  E.g. there is just a hyphen under each of the headings for Status, Last Event, etc.

[franco]

I think it looks like they fixed the server config so can you redo the test in https://github.com/opnsense/plugins/issues/4943#issuecomment-3306314947 ?


Cheers,
Franco

[allenlook]

I redid the test and got an OK result back this time.  The widget does load, but displays a hyphen in each data field instead of the valid dates that seem to be getting passed back by the API.

Thinking it might help to restart the Intrusion Detection service, I did that and the firewall crashed, showing an endless loop of scrolling dump error messages.

I rebooted and everything is running OK at the moment.  I did notice this in the boot messages "pid 31 (zpool) is attempting to use unsafe AIO requests - not logging anymore", and the dmesg.boot file ends on that message.

I did a bunch of searching on that message and the results vary, so at this point I'm going to do nothing, and at least I won't make it worse by pasting commands from the Internet into a working firewall. *nix is an endless series of rabbit holes.  Always has been, since the early days when I was an HP-UX admin many decades ago.

UPDATE: The Suricata log is full of errors from the time of the crash. It shows that the engine restarted, and then within 3 seconds it started posting about 20,000 of these, until I shut it off.

[299151] -- igc0^: error reading netmap data via polling: No error: 0
[299176] -- igc0^: error reading netmap data via polling: No buffer space available

[franco]

I guess there is a bit of confirmation bias at play here. We're debugging a widget, quickly pinpointed the server issue and Proofpoint fixed it. Maybe we're not there 100% yet but to throw in the towel seems like the most discouraging action here towards development and support effort.


Cheers,
Franco

[franco]

I'm taking this to the next dev meeting:

https://github.com/opnsense/plugins/issues/4943#issuecomment-3312623289

Long story short is when the heartbeat couldn't be sent to the server due to server side SSL issue the status went "DORMAT" for users so the widget doesn't display anything by design. We can improve that design, sure, no question, but I think this also is much less important than it was set out to be.


Cheers,
Franco

[allenlook]

OK, thank you.

Is there anything we as users can/should do to get the status from DORMANT back to ACTIVE?

According to the docs, we still receive the Pro rule set, even with DORMANT. https://docs.opnsense.org/manual/etpro_telemetry.html

[franco]

Active status should be back with the next heartbeat acknowledged by Proofpoint.

We're in contact with them to make sure it's back to normal soon.


Cheers,
Franco

[jangw]

--

[allenlook]

I guess there is a bit of confirmation bias at play here. We're debugging a widget, quickly pinpointed the server issue and Proofpoint fixed it. Maybe we're not there 100% yet but to throw in the towel seems like the most discouraging action here towards development and support effort.


Cheers,
Franco

In this case, "doing nothing" means not making the matter worse by executing a bunch of "zpool restore -fingers -crossed" commands that I find on the Internet.  I'll continue to monitor the firewall, watch this thread, and perform whatever commands the experts think might help.