Crowdsec Daemon is stopping at 1am (sometimes)

[mmetc]

Hello,

Thanks for sending logs and configurations, we fixed some issue for the upcoming 1.6.1 and are looking at other possible causes.

In the meantime, we have a version of the base crowdsec package that restarts the service correctly when it fails.

You can find it at https://github.com/crowdsecurity/plugins/releases/tag/crowdsec-1.6.0_3

Let us know if it helps and thanks for testing,

Marco

[opnthib]

Hello,
I think I have the same problem.
The service is stopped, I try to start it, for a few seconds the service icon is green but always returns to red.
I uninstalled - restarted OPNsense - intall Crowdsec, the problem is still there.

Note: I have had the problem for some time.

Code Select Expand


#  tail /var/log/crowdsec/crowdsec-firewall-bouncer.log
time="28-04-2024 22:22:37" level=error msg="Get \"http://127.0.0.1:8080/v1/decisions/stream?\": dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:23:07" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:23:07" level=error msg="Get \"http://127.0.0.1:8080/v1/decisions/stream?\": dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:23:37" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:23:37" level=error msg="Get \"http://127.0.0.1:8080/v1/decisions/stream?\": dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:24:07" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:24:07" level=error msg="Get \"http://127.0.0.1:8080/v1/decisions/stream?\": dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:24:37" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:24:37" level=error msg="Get \"http://127.0.0.1:8080/v1/decisions/stream?\": dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:25:07" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:25:07" level=error msg="Get \"http://127.0.0.1:8080/v1/decisions/stream?\": dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:25:37" level=error msg="auth-api: auth with api key failed return nil response, error: dial tcp 127.0.0.1:8080: i/o timeout"
time="28-04-2024 22:25:37" level=error msg="Get \"http://127.0.0.1:8080/v1/decisions/stream?\": dial tcp 127.0.0.1:8080: i/o timeout"


Code Select Expand


# tail /var/log/crowdsec/crowdsec.log
time="2024-04-28T22:23:30+02:00" level=warning msg="You are using sqlite without WAL, this can have a performance impact. If you do not store the database in a network share, set db_config.use_wal to true. Set explicitly to false to disable this warning."
time="2024-04-28T22:23:30+02:00" level=info msg="Enabled feature flags: <none>"
time="2024-04-28T22:23:30+02:00" level=info msg="Crowdsec v1.6.0-freebsd-4b8e6cd7"
time="2024-04-28T22:23:30+02:00" level=info msg="Loading prometheus collectors"
time="2024-04-28T22:23:31+02:00" level=info msg="Loading CAPI manager"
time="2024-04-28T22:23:32+02:00" level=info msg="CAPI manager configured successfully"
time="2024-04-28T22:23:32+02:00" level=error msg="Machine is not enrolled in the console, can't synchronize with the console"
time="2024-04-28T22:23:32+02:00" level=info msg="CrowdSec Local API listening on 127.0.0.1:8080"
time="2024-04-28T22:23:32+02:00" level=info msg="Start sending metrics to CrowdSec Central API (interval: 23m2s once, then 30m0s)"
time="2024-04-28T22:23:32+02:00" level=info msg="Start push to CrowdSec Central API (interval: 3s once, then 10s)"
time="2024-04-28T22:23:32+02:00" level=info msg="capi metrics: sending"
time="2024-04-28T22:23:32+02:00" level=info msg="last CAPI pull is newer than 1h30, skip."
time="2024-04-28T22:23:32+02:00" level=info msg="Start pull from CrowdSec Central API (interval: 2h1m51s once, then 2h0m0s)"
time="2024-04-28T22:23:32+02:00" level=info msg="Loading grok library /usr/local/etc/crowdsec/patterns"
time="2024-04-28T22:23:34+02:00" level=info msg="Loading enrich plugins"
time="2024-04-28T22:23:34+02:00" level=info msg="Successfully registered enricher 'GeoIpCity'"
time="2024-04-28T22:23:34+02:00" level=info msg="Successfully registered enricher 'GeoIpASN'"
time="2024-04-28T22:23:34+02:00" level=info msg="Successfully registered enricher 'IpToRange'"
time="2024-04-28T22:23:34+02:00" level=info msg="Successfully registered enricher 'reverse_dns'"
time="2024-04-28T22:23:34+02:00" level=info msg="Successfully registered enricher 'ParseDate'"
time="2024-04-28T22:23:34+02:00" level=info msg="Successfully registered enricher 'UnmarshalJSON'"
time="2024-04-28T22:23:34+02:00" level=info msg="Loading parsers from 6 files"
time="2024-04-28T22:23:34+02:00" level=info msg="Loaded 2 parser nodes" file=/usr/local/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml stage=s00-raw
time="2024-04-28T22:23:34+02:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s01-parse/opnsense-gui-logs.yaml stage=s01-parse
time="2024-04-28T22:23:34+02:00" level=info msg="Loaded 2 parser nodes" file=/usr/local/etc/crowdsec/parsers/s01-parse/pf-logs.yaml stage=s01-parse
time="2024-04-28T22:23:34+02:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml stage=s01-parse
time="2024-04-28T22:23:34+02:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml stage=s02-enrich
time="2024-04-28T22:23:34+02:00" level=info msg="Loaded 1 parser nodes" file=/usr/local/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml stage=s02-enrich
time="2024-04-28T22:23:34+02:00" level=info msg="Loaded 8 nodes from 3 stages"
time="2024-04-28T22:23:34+02:00" level=info msg="No postoverflow parsers to load"
time="2024-04-28T22:23:34+02:00" level=info msg="Loading 4 scenario files"
time="2024-04-28T22:23:34+02:00" level=info msg="Adding leaky bucket" cfg=hidden-darkness name=crowdsecurity/opnsense-gui-bf
time="2024-04-28T22:23:34+02:00" level=info msg="Adding leaky bucket" cfg=divine-darkness name=crowdsecurity/ssh-slow-bf
time="2024-04-28T22:23:34+02:00" level=info msg="Adding leaky bucket" cfg=billowing-cloud name=crowdsecurity/ssh-slow-bf_user-enum
time="2024-04-28T22:23:34+02:00" level=info msg="Adding leaky bucket" cfg=icy-voice name=firewallservices/pf-scan-multi_ports
time="2024-04-28T22:23:34+02:00" level=info msg="Adding leaky bucket" cfg=divine-flower name=crowdsecurity/ssh-bf
time="2024-04-28T22:23:34+02:00" level=info msg="Adding leaky bucket" cfg=spring-river name=crowdsecurity/ssh-bf_user-enum
time="2024-04-28T22:23:34+02:00" level=info msg="Loaded 6 scenarios"
time="2024-04-28T22:23:34+02:00" level=info msg="loading acquisition file : /usr/local/etc/crowdsec/acquis.yaml"
time="2024-04-28T22:23:34+02:00" level=warning msg="No matching files for pattern /var/log/nginx/*.log" type=file
time="2024-04-28T22:23:34+02:00" level=warning msg="No matching files for pattern ./tests/nginx/nginx.log" type=file
time="2024-04-28T22:23:34+02:00" level=warning msg="No matching files for pattern /var/log/auth.log" type=file
time="2024-04-28T22:23:34+02:00" level=warning msg="No matching files for pattern /var/log/syslog" type=file
time="2024-04-28T22:23:34+02:00" level=warning msg="No matching files for pattern /var/log/httpd-access.log" type=file
time="2024-04-28T22:23:34+02:00" level=warning msg="No matching files for pattern /var/log/httpd-error.log" type=file
time="2024-04-28T22:23:34+02:00" level=info msg="loading acquisition file : /usr/local/etc/crowdsec/acquis.d/opnsense.yaml"
time="2024-04-28T22:23:34+02:00" level=info msg="Force add watch on /var/log/audit" type=file
time="2024-04-28T22:23:34+02:00" level=info msg="Adding file /var/log/audit/latest.log to datasources" type=file
time="2024-04-28T22:23:34+02:00" level=info msg="Force add watch on /var/log/lighttpd" type=file
time="2024-04-28T22:23:34+02:00" level=info msg="Adding file /var/log/lighttpd/latest.log to datasources" type=file
time="2024-04-28T22:23:34+02:00" level=info msg="Force add watch on /var/log/filter" type=file
time="2024-04-28T22:23:34+02:00" level=info msg="Adding file /var/log/filter/latest.log to datasources" type=file
time="2024-04-28T22:23:34+02:00" level=info msg="Starting processing data"
time="2024-04-28T22:23:34+02:00" level=info msg="Error machine login for  : ent: machine not found "
time="2024-04-28T22:23:34+02:00" level=info msg="retrying in 0 seconds (attempt 2 of 2)"
time="2024-04-28T22:23:34+02:00" level=info msg="Error machine login for  : ent: machine not found "
time="2024-04-28T22:23:34+02:00" level=fatal msg="starting outputs error : authenticate watcher (): API error: ent: machine not found"


[Patrick M. Hausen]

Hopefully going to be fixed with CrowdSec 1.6.1 - ETA "soon" - where they reworked most of the service management.