crowdsec firewall bouncer does not start - pfctl crowdsec-blacklists not exist

[luckylinux]

I installed (or rather attempted to) Crowdsec on the latest OPNSense Release (with all Updates applied: OPNsense 24.1.8-amd64, FreeBSD 13.2-RELEASE-p11, OpenSSL 3.0.13) according to https://docs.crowdsec.net/docs/getting_started/install_crowdsec_opnsense/.

I also enrolled it to the Crowdsec Console (from SSH-ing into my OPNSense Instance).

However, while the Crowdsec Service appears to work correctly, the Firewall Bouncer dies within a second or so after attempting to be started.

OPNSense -> Services -> CrowdSec -> Overview
Service status: crowdsec [tick / success] - firewall bouncer [cross / fail]

Output of `cscli version`:

Code Select Expand

2024/06/04 17:00:55 version: v1.6.1-freebsd-0746e0c0
2024/06/04 17:00:55 Codename: alphaga
2024/06/04 17:00:55 BuildDate: 2024-05-28_00:23:25
2024/06/04 17:00:55 GoVersion: 1.21.10
2024/06/04 17:00:55 Platform: freebsd
2024/06/04 17:00:55 libre2: C++
2024/06/04 17:00:55 Constraint_parser: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_scenario: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_api: v1
2024/06/04 17:00:55 Constraint_acquis: >= 1.0, < 2.0

According to the logs, it seems one Blacklist doesn't exist. Am I supposed to create it manually (it wasn't in the Tutorial), and if so, how ?

OPNSense -> Firewall -> Aliases show that "crowdsec_blacklists" and "crowdsec6_blacklists" exists.
Note the "_" (underscore) instead of the "-" (dash) which pfctl complains in the logs below.

Output of `cat /var/log/crowdsec-firewall-bouncer.log`

Code Select Expand

time="04-06-2024 16:22:55" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:22:55" level=info msg="backend type : pf"
time="04-06-2024 16:22:55" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:22:55" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:22:55" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:22:55" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:34:42" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:34:42" level=info msg="backend type : pf"
time="04-06-2024 16:34:42" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:34:42" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:34:42" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:34:42" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:43" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:43" level=info msg="backend type : pf"
time="04-06-2024 16:50:43" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:43" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:43" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:43" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:47" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:47" level=info msg="backend type : pf"
time="04-06-2024 16:50:47" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:47" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:47" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:47" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:50" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:50" level=info msg="backend type : pf"
time="04-06-2024 16:50:50" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:50" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:50" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:50" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:54:03" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:54:03" level=info msg="backend type : pf"
time="04-06-2024 16:54:03" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:54:03" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:54:03" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:54:03" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:04" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:04" level=info msg="backend type : pf"
time="04-06-2024 16:55:04" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:04" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:04" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:04" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"

[cookiemonster]

I'm on OPN v 22.7 so might not be the right pointer but on it, the table is called crowdsec_blacklists as in your aliases. Seems the code is expecting - instead of _.
Just a guess. Needs crowdsec to advise.

[luckylinux]

I'm on OPN v 22.7 so might not be the right pointer but on it, the table is called crowdsec_blacklists as in your aliases. Seems the code is expecting - instead of _.
Just a guess. Needs crowdsec to advise.

I had the same Impression, but wasn't sure if maybe there is a (uni/bi)directional "_" <-> "-" Conversion happening behind the Scenes, most likely only in one Direction.

[luckylinux]

I am not sure this is the correct Forum/Section for crowdsec. Any Opinion on how to proceed ? I also don't know if this is OPNSense-specific of rather an upstream Issue  :(.

Should I open a BUG Report on the OPNSense Issue Tracker (https://github.com/opnsense/plugins/issues/) ?

[cookiemonster]

sometimes the crowdsec people respond here but best to report directly. They seem to be active on their online thingie that I can't remember what is called. Surely you can get to it from their website.

[cookiemonster]

It's discourse https://discourse.crowdsec.net/

[mmetc]

I installed (or rather attempted to) Crowdsec on the latest OPNSense Release (with all Updates applied: OPNsense 24.1.8-amd64, FreeBSD 13.2-RELEASE-p11, OpenSSL 3.0.13) according to https://docs.crowdsec.net/docs/getting_started/install_crowdsec_opnsense/.

I also enrolled it to the Crowdsec Console (from SSH-ing into my OPNSense Instance).

However, while the Crowdsec Service appears to work correctly, the Firewall Bouncer dies within a second or so after attempting to be started.

OPNSense -> Services -> CrowdSec -> Overview
Service status: crowdsec [tick / success] - firewall bouncer [cross / fail]

Output of `cscli version`:

Code Select Expand

2024/06/04 17:00:55 version: v1.6.1-freebsd-0746e0c0
2024/06/04 17:00:55 Codename: alphaga
2024/06/04 17:00:55 BuildDate: 2024-05-28_00:23:25
2024/06/04 17:00:55 GoVersion: 1.21.10
2024/06/04 17:00:55 Platform: freebsd
2024/06/04 17:00:55 libre2: C++
2024/06/04 17:00:55 Constraint_parser: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_scenario: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_api: v1
2024/06/04 17:00:55 Constraint_acquis: >= 1.0, < 2.0

According to the logs, it seems one Blacklist doesn't exist. Am I supposed to create it manually (it wasn't in the Tutorial), and if so, how ?

OPNSense -> Firewall -> Aliases show that "crowdsec_blacklists" and "crowdsec6_blacklists" exists.
Note the "_" (underscore) instead of the "-" (dash) which pfctl complains in the logs below.

Output of `cat /var/log/crowdsec-firewall-bouncer.log`

Code Select Expand

time="04-06-2024 16:22:55" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:22:55" level=info msg="backend type : pf"
time="04-06-2024 16:22:55" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:22:55" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:22:55" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:22:55" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:34:42" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:34:42" level=info msg="backend type : pf"
time="04-06-2024 16:34:42" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:34:42" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:34:42" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:34:42" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:43" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:43" level=info msg="backend type : pf"
time="04-06-2024 16:50:43" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:43" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:43" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:43" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:47" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:47" level=info msg="backend type : pf"
time="04-06-2024 16:50:47" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:47" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:47" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:47" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:50" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:50" level=info msg="backend type : pf"
time="04-06-2024 16:50:50" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:50" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:50" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:50" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:54:03" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:54:03" level=info msg="backend type : pf"
time="04-06-2024 16:54:03" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:54:03" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:54:03" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:54:03" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:04" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:04" level=info msg="backend type : pf"
time="04-06-2024 16:55:04" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:04" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:04" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:04" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"

The plugin should configure /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml

to use the _ instead of - which is the default value, but not allowed by opnsense.

Can you please run

# configctl crowdsec reconfigure

# tail -f /var/log/configd/latest.log

and see if there's any error?

thanks

[mmetc]

sometimes the crowdsec people respond here but best to report directly. They seem to be active on their online thingie that I can't remember what is called.

Maybe you mean GitHub? :)

[cookiemonster]

ha ha no. I meant discord. You missed my subsequent post :)
Frankly no idea which one is meant to be the official place for support requests. I hope it is Github.

[luckylinux]

The plugin should configure /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml

to use the _ instead of - which is the default value, but not allowed by opnsense.

Can you please run

# configctl crowdsec reconfigure

# tail -f /var/log/configd/latest.log

and see if there's any error?

thanks

Thank you for your Answer.

Here you go:

Code Select Expand

configctl crowdsec reconfigure
OK


Code Select Expand

tail -f /var/log/configd/latest.log
<13>1 2024-06-05T14:42:37+02:00 Router.localdomain configd.py 234 - [meta sequenceId="1"] [b9f126b9-7623-4072-9890-96f072c3d8e0] crowdsec reconfigure
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="2"] [d57dd0fe-b953-4385-96ae-1ec8c01f6d19] Reloading filter
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="3"] [c648db2c-ae47-47a6-9674-c14948d3ba06] request pf current overall table record count and table-entries limit
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="4"] [3bec8a08-36d9-46f4-ab15-bd3111cc8413] list gateways
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="5"] [597c3bf1-f468-4d68-b18a-39e5608a341c] generate template OPNsense/Filter
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="6"] generate template container OPNsense/Filter
<15>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="7"]  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
<15>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="8"]  OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="9"] [e7152f5f-c5b6-481c-b9d5-50aee3779d1d] refresh url table aliases
<14>1 2024-06-05T14:42:41+02:00 Router.localdomain configd.py 234 - [meta sequenceId="10"] message e7152f5f-c5b6-481c-b9d5-50aee3779d1d [] returned b'{"status": "ok"}\n'

Now indeed in OPNSense -> Services -> CrowdSec -> Overview it's indeed better:
Service status: crowdsec [tick / success] - firewall bouncer [tick / success]

But it's still unclear to me why this happens on a stock Install ... and for how long it would even work ???.

[mmetc]

ha ha no. I meant discord. You missed my subsequent post :)
Frankly no idea which one is meant to be the official place for support requests. I hope it is Github.

Discord or Reddit are good for interactive or non-technical support, GitHub for better follow up.

[mmetc]

Now indeed in OPNSense -> Services -> CrowdSec -> Overview it's indeed better:
Service status: crowdsec [tick / success] - firewall bouncer [tick / success]

But it's still unclear to me why this happens on a stock Install ... and for how long it would even work ???.

If there's no error in the reconfigure event, it should keep working. I don't see why it failed the first time.