Hi everyone, new user to Opnsense here. I'm hoping someone can help me shed a light on this problem... I have a token for the ET Pro telemetry rules, and there are events occurring, but the heartbeats are no longer being sent for some reason.
When checking the logs, all I see is the below:
/send_heartbeat.py: unexpected result from https://opnsense.emergingthreats.net/api/v1/telemetry (http_code 404)
Does anyone have any suggestions as to how to fix this? My sensor will eventually go into dormant/disabled if not fixed.
same issue here, looks like the service is down on their side, since several days...
Quote from: siga75 on December 25, 2019, 06:32:00 AM
same issue here, looks like the service is down on their side, since several days...
Thank you for that, glad its not only me. I have tested the name resolution, routing, etc. and everything is correct... 404 indicates not found, so would suspect something is wrong on the other end.
My last heartbeat was Fri Dec 20 20:22:52 +0200 2019
Proofpoint's rule service isn't down (new rules are being served properly it seems), but there might be an issue with their heartbeat stream. I'll send them an email and ask about it.
Quote from: AdSchellevis on December 25, 2019, 12:23:28 PM
Proofpoint's rule service isn't down (new rules are being served properly it seems), but there might be an issue with their heartbeat stream. I'll send them an email and ask about it.
Thank you, I believe you are correct as my rules updated yesterday morning. Just the heartbeats which seem to be an issue.
Thanks!
Yeah, rules get downloaded.
Heartbeat and send_telemetry does not work
They know there's something called "monitoring"?
I mean, the service isn't working since DAYS...
OK, it's free (in exchange for some data, which is fair), OK, rules still get downloaded.
Still this keeps me astonished, where I work I am called during the night (even on holidays) if our monitoring detects issues related to my area (unix). Same for every area of service.
And I wonder now how good can be a service provided by people who takes days to recover a service.
Although it's not our service, I would like comment to on the suggestion that it was down, which isn't correct (as mentioned before).
Rules are being published and delivered properly (as far as I know, I haven't seen proof of anything else).
Complaining about a service (no matter if it's free or paid), without knowledge of the details, feels silly and not very constructive. For all you know Proofpoint might monitor crucial areas of the service actively and non vital parts less frequent, issues might occur only in specific scenarios, etc, etc.... just saying, context is important.
Best regards,
Ad
Quote from: AdSchellevis on December 26, 2019, 08:53:19 PM
Complaining about a service (no matter if it's free or paid), without knowledge of the details, feels silly and not very constructive. For all you know Proofpoint might monitor crucial areas of the service actively and non vital parts less frequent, issues might occur only in specific scenarios, etc, etc.... just saying, context is important.
It's not silly and I am not complaining, it was just a consideration.
They are loosing a huge amount of "precious" data, if this is not crucial for them, maybe this data are not so well investigated, and this makes me wonder about the quality of the signatures.
I don't know the details simply because they even didn't send an email, and they should have warned us, since we have the logs full with entries every 1 minute
Not going to spend more time on this, but how would they send you an email if they don't know who you are... https://docs.opnsense.org/manual/etpro_telemetry.html
If I read the error message correctly, the endpoint in your case isn't about actual statistics gathering (which you can easily validate by reading the also public source).
Jumping to conclusions (about the quality of signatures) so easily is again not very constructive and considered harmful for those involved.
Best regards,
Ad
Quote from: AdSchellevis on December 27, 2019, 09:39:21 AM
Not going to spend more time on this, but how would they send you an email if they don't know who you are... https://docs.opnsense.org/manual/etpro_telemetry.html
If I read the error message correctly, the endpoint in your case isn't about actual statistics gathering (which you can easily validate by reading the also public source).
Jumping to conclusions (about the quality of signatures) so easily is again not very constructive and considered harmful for those involved.
Best regards,
Ad
For what it is worth, I am in full agreement with you - the service itself is working, rules are being served etc., just heartbeats. I am not entirely sure as to where the heartbeats fit into the picture - would I be correct in saying that they allow ET to see which sensors are active/streaming?
Additional question: was I correct to report this here, or should I have reached out to ET directly?
it's no problem at all to report it here, we can pass the message if needed.
Quote from: AdSchellevis on December 27, 2019, 09:39:21 AM
Not going to spend more time on this, but how would they send you an email if they don't know who you are... https://docs.opnsense.org/manual/etpro_telemetry.html
If I read the error message correctly, the endpoint in your case isn't about actual statistics gathering (which you can easily validate by reading the also public source).
Jumping to conclusions (about the quality of signatures) so easily is again not very constructive and considered harmful for those involved.
Best regards,
Ad
I will also stop wasting time, I think my point was not so hard to understand...
BTW: they have my email, since I registered
Quote from: AdSchellevis on December 27, 2019, 12:53:16 PM
it's no problem at all to report it here, we can pass the message if needed.
Thank you Ad, I appreciate the guidance.
Update: looks like it is working again
Last heartbeat
Fri Dec 27 21:28:58 +0200 2019
hearthbeat is working now, send telemetry is still not working
[root@myfw ~]# time /usr/local/opnsense/scripts/etpro_telemetry/send_telemetry.py
real 0m23.418s
user 0m0.367s
sys 0m0.048s
[root@myfw ~]# echo $?
255
Dec 28 06:31:00 /send_telemetry.py: telemetry data collected 1 records in 0.06 seconds @2019-12-28 01:18:01.170855
Dec 28 06:30:00 /send_telemetry.py: telemetry data collected 1 records in 0.07 seconds @2019-12-28 01:18:01.170855
Dec 28 06:29:00 /send_telemetry.py: telemetry data collected 1 records in 0.06 seconds @2019-12-28 01:18:01.170855
Dec 28 06:28:00 /send_telemetry.py: telemetry data collected 1 records in 0.06 seconds @2019-12-28 01:18:01.170855
Dec 28 06:27:00 /send_telemetry.py: telemetry data collected 1 records in 0.09 seconds @2019-12-28 01:18:01.170855
Dec 28 06:26:00 /send_telemetry.py: telemetry data collected 1 records in 0.06 seconds @2019-12-28 01:18:01.170855
Dec 28 06:25:00 /send_telemetry.py: telemetry data collected 1 records in 0.20 seconds @2019-12-28 01:18:01.170855
Dec 28 06:24:00 /send_telemetry.py: telemetry data collected 1 records in 0.06 seconds @2019-12-28 01:18:01.170855
Dec 28 06:23:00 /send_telemetry.py: telemetry data collected 1 records in 0.07 seconds @2019-12-28 01:18:01.170855
Dec 28 06:22:00 /send_telemetry.py: telemetry data collected 1 records in 0.05 seconds @2019-12-28 01:18:01.170855
Dec 28 06:21:00 /send_telemetry.py: telemetry data collected 1 records in 0.06 seconds @2019-12-28 01:18:01.170855
Dec 28 06:20:00 /send_telemetry.py: telemetry data collected 1 records in 0.20 seconds @2019-12-28 01:18:01.170855
Dec 28 06:19:00 /send_telemetry.py: telemetry data collected 1 records in 0.05 seconds @2019-12-28 01:18:01.170855
Dec 28 06:18:00 /send_telemetry.py: telemetry data collected 1 records in 0.23 seconds @2019-12-28 01:18:01.170855
Dec 28 06:17:00 /send_telemetry.py: telemetry data collected 1 records in 0.06 seconds @2019-12-28 01:18:01.170855
Dec 28 06:16:00 /send_telemetry.py: telemetry data collected 1 records in 0.15 seconds @2019-12-28 01:18:01.170855
Dec 28 06:15:00 /send_telemetry.py: telemetry data collected 1 records in 0.16 seconds @2019-12-28 01:18:01.170855
Dec 28 06:14:00 /send_telemetry.py: telemetry data collected 1 records in 0.06 seconds @2019-12-28 01:18:01.170855
It is working (the same it has been doing for quite a while).
https://github.com/opnsense/plugins/issues/1642
NOW it's working, this morning I got a timeout after more than 20 second.
[root@myfw ~]# time /usr/local/opnsense/scripts/etpro_telemetry/send_telemetry.py
real 0m0.292s
user 0m0.269s
sys 0m0.024s
As it has been doing all along, assumptions are usually not very valuable.
# /usr/local/opnsense/scripts/etpro_telemetry/send_telemetry.py -h
usage: send_telemetry.py [-h] [-e ENDPOINT] [-i] [-c CONFIG] [-l LOG]
[-s STATE] [-d DAYS] [-D]
optional arguments:
-h, --help show this help message and exit
-e ENDPOINT, --endpoint ENDPOINT
Endpoint url to reach
-i, --insecure Insecure, skip certificate validation
-c CONFIG, --config CONFIG
rule downloader configuration
-l LOG, --log LOG log directory containing eve.json files
-s STATE, --state STATE
persistent state (and lock) filename
-d DAYS, --days DAYS Maximum number of days to look back on initial run
-D, --direct do not sleep before send (disable traffic spread) <-------------------- !!
LOL I don't get why do you insist so much...
This morning it was not working, the command I executed is exactly the same, and it's what is in crontab.
Now it works
[root@myfw ~]# cat /usr/local/etc/cron.d/etpro-telemetry.cron
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
#
# User-defined crontab files can be loaded via /etc/cron.d
# or /usr/local/etc/cron.d and follow the same format as
# /etc/crontab, see the crontab(5) manual page.
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
#minute hour mday month wday who command
0 * * * * root /usr/local/opnsense/scripts/etpro_telemetry/send_heartbeat.py
* * * * * root /usr/local/opnsense/scripts/etpro_telemetry/send_telemetry.py
[root@myfw ~]# /usr/local/opnsense/scripts/etpro_telemetry/send_telemetry.py
[root@myfw ~]# echo $?
0
Compare this with the 23 seconds when IT WAS NOT WORKING
[root@myfw ~]# while true; do time /usr/local/opnsense/scripts/etpro_telemetry/send_telemetry.py; done
real 0m0.351s
user 0m0.312s
sys 0m0.040s
real 0m0.354s
user 0m0.293s
sys 0m0.062s
real 0m0.356s
user 0m0.265s
sys 0m0.092s
real 0m0.355s
user 0m0.317s
sys 0m0.040s
real 0m0.341s
user 0m0.278s
sys 0m0.063s
real 0m0.344s
user 0m0.299s
sys 0m0.046s
real 0m0.352s
user 0m0.297s
sys 0m0.056s
real 0m0.348s
user 0m0.279s
sys 0m0.070s
real 0m0.382s
user 0m0.342s
sys 0m0.040s
real 0m0.352s
user 0m0.330s
sys 0m0.024s
real 0m0.376s
user 0m0.322s
sys 0m0.055s
real 0m0.370s
user 0m0.308s
sys 0m0.063s
real 0m0.360s
user 0m0.321s
sys 0m0.040s
real 0m0.339s
user 0m0.285s
sys 0m0.055s
real 0m1.483s
user 0m0.494s
sys 0m0.031s
real 0m0.569s
user 0m0.490s
sys 0m0.064s
real 0m0.474s
user 0m0.392s
sys 0m0.063s
real 0m0.341s
user 0m0.318s
sys 0m0.023s
real 0m0.346s
user 0m0.300s
sys 0m0.048s
real 0m0.362s
user 0m0.285s
sys 0m0.078s
^C
Do you think I have no idea what I am speaking about, you are totally wrong
From where does YOUR "assumption" that it was working? You even didn't notice the issue before we post the case here. I did several test, and I tell you it was not working
Now I will stop reply since it's becoming ridiculous, I just posted a consideration in first post, no need to defend Telemetry
Only for reference, because code doesn't lie.
https://github.com/opnsense/plugins/blob/9c4b22815d1d4b2b954d84fae28f9f8c172cb27d/security/etpro-telemetry/src/opnsense/scripts/etpro_telemetry/send_telemetry.py#L87-L88
(random wait between 0 and 60 seconds, when not supplying the parameter in my previous posting and there's data to be processed)