OPNsense Forum

English Forums => 24.7, 24.10 Legacy Series => Topic started by: ivwang on January 19, 2025, 06:10:20 AM

Title: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: ivwang on January 19, 2025, 06:10:20 AM
Hi all,

After 24.7.12 upgrade, it looks like the ET Pro Telemetry widget shows either empty box or "failed to load widget".

From the CLI, sensor_info.py *sometimes* works, and when it works it takes several seconds to complete, though it reports the sensor is still ACTIVE, while other times the script failed with exception stating remote end closed connection, like below:

--------
  File "/usr/local/lib/python3.11/http/client.py", line 294, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
--------

Anyone also seeing this?

Thanks a lot.
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: RamSense on January 19, 2025, 08:46:41 AM
yes here the same, see also here: https://forum.opnsense.org/index.php?topic=45112.0 (https://forum.opnsense.org/index.php?topic=45112.0)
and here: https://forum.opnsense.org/index.php?topic=40751.0 (https://forum.opnsense.org/index.php?topic=40751.0)
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: RES217AIII on January 19, 2025, 04:59:58 PM
I have the same problem
Is this possibly the answer to the problem?

https://community.emergingthreats.net/t/etpro-telemetry-edition/2355 (https://community.emergingthreats.net/t/etpro-telemetry-edition/2355)

"For sensors opting-in to sending Proofpoint/ET telemetry so they can receive ETPRO telemetry edition those sensors must have sent event telemetry back to Proofpoint/ET within the last 5 days.

Sensors may go dormant during that period (no heartbeat sent in the last day) and still receive ETPRO Telemetry Edition, but if no events are received for 5 days the Telemetry Edition rule delivery will be disabled and that sensor will simply receive that day's ET Open rules.

That disabling will transition back to active delivery upon resumption of heartbeat and telemetry delivery back to Proofpoint/ET.

Sensors are reviewed as to state every 24 hours."
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: RamSense on January 19, 2025, 06:11:00 PM
I have installed a new opnsense box and ordered a new token for ET Pro Telementry, and the same result with the widget.
So maybe there is something with the widget or with opnsense not sending the heartbeat??
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: irrenarzt on January 19, 2025, 06:59:53 PM
Same issue, but I was starting to have problems before 24.7.12. The token was expiring after approximately 5 days despite consistent heartbeats. I contacted ET Labs and they said they've received multiple reports of this and were looking into it.

Something has further degraded though... just like you guys, updating the token isn't fixing the issue anymore. I can't update ET Pro rules, the widget doesn't work, and I'm getting the same error in my logs as above. That said, I'm confident it's not an OPNsense issue.
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: sbaran50 on January 20, 2025, 09:54:41 AM
Exactly the same issue here, STATUS_PY says its ACTIVE.
But mine struggles to send python3 send_heartbeat.py.
Its doing nothing for 10 minutes, last HB was sent yesterday, after i have completely reinstalled the FW from scratch.
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: jglatz on January 20, 2025, 03:18:38 PM
I am having the same issue on 24.10.1, been flaky for a month or so. I've had to order new tokens to get it working. Starting in the last week, I am seeing the exact same thing. Widget won't load, invoking the sensor_info.py command takes a very long time when it works. Most of the time it will fail
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: klamath on January 20, 2025, 05:26:49 PM
24.10.1-amd64 same issue:

  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: Monviech (Cedrik) on January 20, 2025, 05:28:57 PM
The issue is tracked here:

https://github.com/opnsense/plugins/issues/4486

We will respond there with new info when we have it.
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: corran22 on January 22, 2025, 11:55:35 PM
Quote from: RES217AIII on January 19, 2025, 04:59:58 PMI have the same problem
Is this possibly the answer to the problem?

https://community.emergingthreats.net/t/etpro-telemetry-edition/2355 (https://community.emergingthreats.net/t/etpro-telemetry-edition/2355)

"For sensors opting-in to sending Proofpoint/ET telemetry so they can receive ETPRO telemetry edition those sensors must have sent event telemetry back to Proofpoint/ET within the last 5 days.

Sensors may go dormant during that period (no heartbeat sent in the last day) and still receive ETPRO Telemetry Edition, but if no events are received for 5 days the Telemetry Edition rule delivery will be disabled and that sensor will simply receive that day's ET Open rules.

That disabling will transition back to active delivery upon resumption of heartbeat and telemetry delivery back to Proofpoint/ET.

Sensors are reviewed as to state every 24 hours."


Greetings - we've modified the token code to re-enable sensors which had been disabled in this period as well as open up the window that's examined to determine whether a sensor is still sending us data (or not). Apologies for the disruption. We'll get some documentation out clarifying our position on telemetry reception and periodicy soon.--ET Team
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: RayonRa on January 23, 2025, 06:32:01 PM
Hi,
i still have the issue.
https://github.com/opnsense/plugins/issues/4486#issuecomment-2608435385
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: corran22 on January 24, 2025, 07:44:48 PM
We're looking at the 502 errors - this is not a sensor disabling issue.  Resource-wise our internal monitoring shows the server healthy and responsive.  Are you still having this problem?
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: RayonRa on January 25, 2025, 10:52:09 AM
Quote from: corran22 on January 24, 2025, 07:44:48 PMWe're looking at the 502 errors - this is not a sensor disabling issue.  Resource-wise our internal monitoring shows the server healthy and responsive.  Are you still having this problem?
Yes, i still have the problem.

2025-01-25T10:50:52   Error   send_telemetry.py   unexpected result from https://opnsense.emergingthreats.net/api/v1/event (http_code 502)   
2025-01-25T10:49:11   Error   send_telemetry.py   unexpected result from https://opnsense.emergingthreats.net/api/v1/event (http_code 502)   
2025-01-25T10:48:24   Error   send_telemetry.py   unexpected result from https://opnsense.emergingthreats.net/api/v1/event (http_code 502)


I tried a rule updater:

2025-01-25T11:09:49   Error   rule-updater.py   download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/version (http_code: 502)   
2025-01-25T11:09:48   Error   rule-updater.py   download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/version (http_code: 502)   
2025-01-25T11:09:48   Error   rule-updater.py   download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/engine/suricata/5 (http_code: 502)   
2025-01-25T11:09:47   Error   rule-updater.py   download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/version (http_code: 502)   
2025-01-25T11:09:47   Error   rule-updater.py   download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/engine/suricata/5 (http_code: 502)
Title: Re: ET Pro Telemetry widget fail to load/empty with 24.7.12
Post by: RayonRa on January 29, 2025, 08:54:26 PM
After a reboot the 502 is gone.
I'm able to download the rules.
BUT (see screenshot)
And i can't see send_telemetry.py work.