Messages

Hi,

this is a custom scenario you made, so I can't exactly replicate. Can you send the content of scenario.yaml to support@crowdsec.net? Attaching the output of "cscli support dump" can help as well. Thanks!

Hello,

the way to debug the issue is to look for the reason in /var/log/crowdsec/crowdsec.log

if you still don't see why it would restart, you can run "cscli support dump" and send the resulting file to support@crowdsec.net

Hi, I'm the plugin maintainer and was not able to reproduce the behavior.

If you had issues with the service start/stop during the package upgrade or at any other time, it would help if you run "cscli support dump" and send the output to support@crowdsec.net. It includes log files and part of the configuration (passwords removed ofc).

A look at that could also explain the initial errors of "cscli ... list", due to the service not running.
Thanks!

You can find the official crowdsec thread on this issue here:

https://discourse.crowdsec.net/t/bug-opnsense-24-7-5-crowdsec-1-6-3/2057

I have executed the command "killall crowdsec".
12834 of 13158 are processes by the name "notification-*".

How can I stop these processes?

"kill 12834" and keep the most recent.

Why does this problem occur?

tl;dr my fault, longer version: each freebsd package does service management in a slightly different way: start, stop, restart if error but not too often, reload configuration, coordinate process groups... there is no unified way to express the application's needs, like the - admittedly not universally popular - systemd system in linux. Which means it requires more scripts to manage corner cases, and more room for errors.

Hi,

could you test this

# fetch -o /usr/local/etc/rc.d/crowdsec https://github.com/crowdsecurity/plugins/releases/download/crowdsec-1.6.3-2-hotfix/crowdsec

and try start/stop.

Thanks

Do I have to use an additional command to install the hotfix?
I suspect that the update did not work. (Screenshots attached)

No it's ok. The fetch command overwrites a script without installing a new package version. Now if you click start/stop from the UI it should just work.

Thank you for the information.

I have noticed that the service can be stopped via the GUI. (Visible because service status is deactivated in the crowdsec overview)
However, the service is displayed as active in the dashboard and in the overview of services.

You have orphan crowdsec processes and possibly notification plugins.

"killall crowdsec" and check if there are processes that go by the name "notification-*"

Hi,

could you test this

# fetch -o /usr/local/etc/rc.d/crowdsec https://github.com/crowdsecurity/plugins/releases/download/crowdsec-1.6.3-2-hotfix/crowdsec

and try start/stop.

Thanks

Do I have to use an additional command to install the hotfix?
I suspect that the update did not work. (Screenshots attached)

No it's ok. The fetch command overwrites a script without installing a new package version. Now if you click start/stop from the UI it should just work.

Hi,

could you test this

# fetch -o /usr/local/etc/rc.d/crowdsec https://github.com/crowdsecurity/plugins/releases/download/crowdsec-1.6.3-2-hotfix/crowdsec

and try start/stop.

Thanks

Hi, crowdsec maintainer here

First thing, the daemon manager had an issue and ignored the INT signal sometimes, in this case the upgrade tries to stop the service and fails.

root    40599  4.0  6.8 1390784 104344  -  S    09:38   0:01.64 /usr/local/bin/crowdsec -c /usr/local/etc/crowdsec/config.yaml
root    40515  0.0  0.1   12736   2164  -  Ss   09:38   0:00.00 daemon: crowdsec[40599] (daemon)

terminate the second process (kill -9 40515) and upgrade to 1.6.3. This changes the script to send a "stronger" signal to stop the process.

Now to understand why it happened, it would be helpful if you could run "cscli support dump" and send the result to support@crowdsec.net. Let us know if, after the upgrade, you still think crowdsec uses too much cpu or ram. It's not a lightweight process but it should not trigger monitoring.

Thanks

Hi, I'm the plugin maintainer

This can happen with 1.6.2 when crowdsec is misbehaving for other reasons (possibly misconfigured, port not available etc) then the daemon manager won't stop it correctly because it's stuck in a restart loop. The opnsense upgrade should install 1.6.3 which fixed the issue by using a "stronger" signal to terminate the daemon manager.

I suggest kill -9, or even the broader

```
# kill -9 `ps xw | grep crowdsec | grep -v grep | awk '{print $1'}`
```

then update to 1.6.3, but should happen with the opnsense upgrade


If there is an underlying issue preventing crowdsec from working correctly, the most "complete" way to report it is via "cscli support dump" and email the resulting file to support@crowdsec.net. This includes logs and part of the configuration so it saves us time asking for details.

Now indeed in OPNSense -> Services -> CrowdSec -> Overview it's indeed better:
Service status: crowdsec [tick / success] - firewall bouncer [tick / success]

But it's still unclear to me why this happens on a stock Install ... and for how long it would even work ???.

If there's no error in the reconfigure event, it should keep working. I don't see why it failed the first time.

ha ha no. I meant discord. You missed my subsequent post :)
Frankly no idea which one is meant to be the official place for support requests. I hope it is Github.

Discord or Reddit are good for interactive or non-technical support, GitHub for better follow up.

sometimes the crowdsec people respond here but best to report directly. They seem to be active on their online thingie that I can't remember what is called.

Maybe you mean GitHub? :)

I installed (or rather attempted to) Crowdsec on the latest OPNSense Release (with all Updates applied: OPNsense 24.1.8-amd64, FreeBSD 13.2-RELEASE-p11, OpenSSL 3.0.13) according to https://docs.crowdsec.net/docs/getting_started/install_crowdsec_opnsense/.

I also enrolled it to the Crowdsec Console (from SSH-ing into my OPNSense Instance).

However, while the Crowdsec Service appears to work correctly, the Firewall Bouncer dies within a second or so after attempting to be started.

OPNSense -> Services -> CrowdSec -> Overview
Service status: crowdsec [tick / success] - firewall bouncer [cross / fail]

Output of `cscli version`:

Code Select Expand

2024/06/04 17:00:55 version: v1.6.1-freebsd-0746e0c0
2024/06/04 17:00:55 Codename: alphaga
2024/06/04 17:00:55 BuildDate: 2024-05-28_00:23:25
2024/06/04 17:00:55 GoVersion: 1.21.10
2024/06/04 17:00:55 Platform: freebsd
2024/06/04 17:00:55 libre2: C++
2024/06/04 17:00:55 Constraint_parser: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_scenario: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_api: v1
2024/06/04 17:00:55 Constraint_acquis: >= 1.0, < 2.0

According to the logs, it seems one Blacklist doesn't exist. Am I supposed to create it manually (it wasn't in the Tutorial), and if so, how ?

OPNSense -> Firewall -> Aliases show that "crowdsec_blacklists" and "crowdsec6_blacklists" exists.
Note the "_" (underscore) instead of the "-" (dash) which pfctl complains in the logs below.

Output of `cat /var/log/crowdsec-firewall-bouncer.log`

Code Select Expand

time="04-06-2024 16:22:55" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:22:55" level=info msg="backend type : pf"
time="04-06-2024 16:22:55" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:22:55" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:22:55" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:22:55" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:34:42" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:34:42" level=info msg="backend type : pf"
time="04-06-2024 16:34:42" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:34:42" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:34:42" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:34:42" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:43" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:43" level=info msg="backend type : pf"
time="04-06-2024 16:50:43" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:43" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:43" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:43" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:47" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:47" level=info msg="backend type : pf"
time="04-06-2024 16:50:47" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:47" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:47" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:47" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:50" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:50" level=info msg="backend type : pf"
time="04-06-2024 16:50:50" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:50" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:50" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:50" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:54:03" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:54:03" level=info msg="backend type : pf"
time="04-06-2024 16:54:03" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:54:03" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:54:03" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:54:03" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:04" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:04" level=info msg="backend type : pf"
time="04-06-2024 16:55:04" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:04" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:04" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:04" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"

The plugin should configure /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml

to use the _ instead of - which is the default value, but not allowed by opnsense.

Can you please run

# configctl crowdsec reconfigure

# tail -f /var/log/configd/latest.log

and see if there's any error?

thanks

If you don't want to reinstall you can remove the machine, remove login and password from /usr/local/etc/crowdsec/local_api_credentials.yaml and restart the services, that should fix it but I have not tried.

this way didn't work, but the removal of db and config folder worked thanks.

this plugin and ntopng are high maintenance, and sort of unreliable. crowdsec GUI also always lies everything is OK. the alias is getting empty, i have a scheduled task to restart crowdsec every few hours. i thought i was out of woods, and stopped the months long stressful watching, but today i noticed the blocklist is again empty. checked the logs, and it didn't load for 2months! i don't know why, but i've also got the same issue here.

Hi, if you are running the latest version could you please send to support@crowdsec.net:

- the result of "cscli support dump"
- the content of /var/log/crowdsec
- the output of "sed -n '/<crowdsec>/,/<\/crowdsec>/p' /conf/config.xml"

I'd like to get to the bottom of this, thanks!