Messages

I installed (or rather attempted to) Crowdsec on the latest OPNSense Release (with all Updates applied: OPNsense 24.1.8-amd64, FreeBSD 13.2-RELEASE-p11, OpenSSL 3.0.13) according to https://docs.crowdsec.net/docs/getting_started/install_crowdsec_opnsense/.

I also enrolled it to the Crowdsec Console (from SSH-ing into my OPNSense Instance).

However, while the Crowdsec Service appears to work correctly, the Firewall Bouncer dies within a second or so after attempting to be started.

OPNSense -> Services -> CrowdSec -> Overview
Service status: crowdsec [tick / success] - firewall bouncer [cross / fail]

Output of `cscli version`:

Code Select Expand

2024/06/04 17:00:55 version: v1.6.1-freebsd-0746e0c0
2024/06/04 17:00:55 Codename: alphaga
2024/06/04 17:00:55 BuildDate: 2024-05-28_00:23:25
2024/06/04 17:00:55 GoVersion: 1.21.10
2024/06/04 17:00:55 Platform: freebsd
2024/06/04 17:00:55 libre2: C++
2024/06/04 17:00:55 Constraint_parser: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_scenario: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_api: v1
2024/06/04 17:00:55 Constraint_acquis: >= 1.0, < 2.0

According to the logs, it seems one Blacklist doesn't exist. Am I supposed to create it manually (it wasn't in the Tutorial), and if so, how ?

OPNSense -> Firewall -> Aliases show that "crowdsec_blacklists" and "crowdsec6_blacklists" exists.
Note the "_" (underscore) instead of the "-" (dash) which pfctl complains in the logs below.

Output of `cat /var/log/crowdsec-firewall-bouncer.log`

Code Select Expand

time="04-06-2024 16:22:55" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:22:55" level=info msg="backend type : pf"
time="04-06-2024 16:22:55" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:22:55" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:22:55" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:22:55" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:34:42" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:34:42" level=info msg="backend type : pf"
time="04-06-2024 16:34:42" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:34:42" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:34:42" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:34:42" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:43" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:43" level=info msg="backend type : pf"
time="04-06-2024 16:50:43" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:43" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:43" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:43" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:47" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:47" level=info msg="backend type : pf"
time="04-06-2024 16:50:47" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:47" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:47" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:47" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:50" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:50" level=info msg="backend type : pf"
time="04-06-2024 16:50:50" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:50" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:50" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:50" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:54:03" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:54:03" level=info msg="backend type : pf"
time="04-06-2024 16:54:03" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:54:03" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:54:03" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:54:03" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:04" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:04" level=info msg="backend type : pf"
time="04-06-2024 16:55:04" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:04" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:04" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:04" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"

The plugin should configure /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml

to use the _ instead of - which is the default value, but not allowed by opnsense.

Can you please run

# configctl crowdsec reconfigure

# tail -f /var/log/configd/latest.log

and see if there's any error?

thanks

If you don't want to reinstall you can remove the machine, remove login and password from /usr/local/etc/crowdsec/local_api_credentials.yaml and restart the services, that should fix it but I have not tried.

this way didn't work, but the removal of db and config folder worked thanks.

this plugin and ntopng are high maintenance, and sort of unreliable. crowdsec GUI also always lies everything is OK. the alias is getting empty, i have a scheduled task to restart crowdsec every few hours. i thought i was out of woods, and stopped the months long stressful watching, but today i noticed the blocklist is again empty. checked the logs, and it didn't load for 2months! i don't know why, but i've also got the same issue here.

Hi, if you are running the latest version could you please send to support@crowdsec.net:

- the result of "cscli support dump"
- the content of /var/log/crowdsec
- the output of "sed -n '/<crowdsec>/,/<\/crowdsec>/p' /conf/config.xml"

I'd like to get to the bottom of this, thanks!

Thanks! Quick fix and PR in

https://github.com/opnsense/plugins/issues/3985#issuecomment-2118451676

Hello,

Thanks for sending logs and configurations, we fixed some issue for the upcoming 1.6.1 and are looking at other possible causes.

In the meantime, we have a version of the base crowdsec package that restarts the service correctly when it fails.

You can find it at https://github.com/crowdsecurity/plugins/releases/tag/crowdsec-1.6.0_3

Let us know if it helps and thanks for testing,

Marco

Yes crowdsec would inappropriately raise an error if a watched file disappears immediately after the initial directory scan
This will be corrected for 1.6.1, but I'm not sure how often it occurs.
More generally, a process exit by crowdsec could be due to CAPI being unavailable for a long time or other issues.

On the linux package any transient exit/crash is not a problem, expect for the possible underlying bug, since the process is restarted immediately by systemd (or docker). For freebsd there is no - afaik - general consensus on how to restart crashed processes.

Monit is a good solution but it's not available on freebsd by default or in pfsense. I tried simply adding a restart option to the sbin/daemon wrapper, it's not working as expected but I'd prefer the solution should be the same for the three platforms.

If someone is using monit to restart crowdsec, can you share that part of configuration?

Thanks

I'm also having this issue. I received an email from Maxmind yesterday stating they would be switching to R2 presigned URLs for all DBs, as of May 1st, and that it is a potential breaking change. Not sure if this is related to the issue we are facing but I figured I would mention it.

I tried running "cscli hub upgrade --force" on both of my routers and they fail on the "crowdsecurity/geoip-enrich" list.

Did you run "cscli hub update" first?

I could not replicate the issue, but it would help if you ran "cscli support dump" and send the resulting file to support@crowdsec.net

Thanks!

I've had to spend most of the weekend fixing my network for other reasons.
Those error messages seem pretty serious and seems MaxMid's database is in a different to the expected. As to what changed would be a guess. Can be either maxmind or crowdsec.

Hi, I'm the author of the opnsense plugin. A new version of the geoip database had issues with the current crowdsec and we reverted to the older version. Hub upgrade (manually or from cron) fixes it, and I don't think it could crash the service. I am looking into the issue. Thanks!

Hi,

it's not the bouncer logs that you should read, but crowdsec.log

Is there anything that points to a service failure?

Hi mmetc

Firstly, OPNsense and CrowdSec seem both to be operating happily now.

The rule I had added (and now abandoned pending testing) blocked internal clients from accessing external sites in the crowdsec_blocklist. I did not have .CW in the rule myself.

I have logging of CrowdSec-initiated block events switched on, and events tagged with .CW in the log. That has been there all along and seems to cause no problem, nor would I expect it to given logs are written and forgotten.

The fix was I completely removed the CrowdSec package then reinstalled it from the repository.

Thanks for the update, that's definitely curious. I could trigger a syntax error by creating a ".CW" tag but if it gave no issue and you don't know where it came from...
I'll do some more testing, thanks again

Hi, I'm the plugin maintainer.

What happens is simply that the bouncer does not wait for the lapi service to be responsive, and is not restarted automatically. Restarting only the bouncer is correct. Restarting both may work or not.

You can try a new version here - https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.28-rc5/crowdsec-firewall-bouncer-0.0.28.r5.pkg

We are releasing it with crowdsec 1.5.3, but it will take a few day to land in the freebsd and opnsense repositories.

I'd be glad if you could confirm that the above version is working for you.

still not working for me with a plugin or even a full reboot.

Hi, could you try "service crowdsec_firewall restart" ?
Otherwise I should have a look at the last few lines in /var/log/crowdsec/crowdsec-firewall-bouncer.log

Hi!

Did you recently add the tag ".CW" by chance? It seems like dots are not allowed in rule tags. This creates a syntax error in the rule file, and they are all loaded together by pfctl design so nothing works.

I'll validate the form field in the next version. Does it work for you if you change or remove the tag?

Hi!

Unfortunately, there is a one-line change required to have crowdsec 1.5+ pick up logs in opnsense. The release was tested with regular files but not symlinks.

You may not notice if you have additional scenarios and agents that don't acquire logs from symlinks, which is why for some people it's working.

The change is in /usr/local/etc/crowdsec/acquis.d/opnsense.yaml, just after force_inotify: true:

poll_without_inotify: true

followed by "# service crowdsec reload" or restart from the GUI

The fix has been merged in version 1.0.6 of the plugin.

Hi, thanks for the report

I could not replicate the issue, downgrading to 1.0.3 - 1.4.6 - 0.0.23.rc2, they all updated and didn't require kill or reboot.

Between 0.0.23.rc2 and 0.0.27 the ip removal is a lot (100x) faster and we did a reworking of the concurrency and signal management so I strongly doubt the issue would happen again.

For 0.0.23.rc2 I could update the plugin to do the kill -9, but I thought that the "service" command would already do that.
I suspect the bouncer could be slow while removing banned ips one by one, so it would be harder to replicate on a fast machine or vm. I'll try with 200k+ decisions.

Hello!

Which version of the plugin are you using?

Can you please check from the console "cscli machines list" - and the last heartbeat. If you have only one server, you should see only one machine.

crowdsec has two parts - a client and a server, in the same executable. they talk through http. the column "name" in machines list should match the login value in /usr/local/etc/crowdsec/local_api_credentials.yaml. If they match, the password is wrong for some reason. Which I'd like to know -- for example in some nas hardware I've seen the random generator behave in a strange way.

Anyway, you don't need a running crowdsec to reset the password.

# cscli machines delete <machine-id>
# rm /usr/local/etc/crowdsec/local_api_credentials.yaml
# umask 077; cscli machines add --auto

and restart the service. If it still does not work, try providing an explicit password instead of --auto, and let me know