How-To crowdsec protecting haproxy

[cookiemonster]

I have been meaning to write this small how-to for a little while. It is working for me and I will not pretend to be fully understanding of Crowdsec's ins and outs. It wouldn't surprise me if there are incorrect assumptions or errors in my understanding. No warranties given but with that out of the way, first some context.
I have a handful of services that I expose to the open internet but only for me to use. These services are hosted on my LAN. I use layers of security and is unusual for me to use simple port forwards. Instead, services are usually behind a reverse proxy (haproxy) which sits on OPNSense, plus the usual additional protections like fail2ban and other methods.
This how-to is to add crowdsec captcha protection to haproxy on OPNSense, specifically to the haproxy plugin. It might have been possible to get crowdsec to read the httpd-access and httpd-error logs from Apache on the freeBSD jail it is protecting but I wanted to separate the tests I was doing at the time, creating and destroying jails frequently. With my preferred setup, crowdsec is protecting all services behind all jails in a central place. Additionally I don't have to setup crowdsec on each target jail, virtual machine or host accessed via haproxy.
The simplified logical setup is as follows:

You cannot view this attachment.
I already have a crowdsec distributed setup link1, where the LAPI runs on OPN, using the crowdsec plugin. The plugin installs the default crowdsec firewall bouncer and other components that are part of the crowdsec/opnsense collection. For this how-to, it is assumed that crowdsec has been setup on OPN already and is working  with the LAPI enabled. There is no need for a distributed setup but it will work just fine in a distributed or standalone setup. Note 1: for clarity, the link to the distributed setup is one I used to base my setup but I did it differently to the example where my setup is reversed from the example i.e. the public machine has the LAPI.

I have haproxy on OPN setup as per the @thehellsite's guide link2. I have a public wildcard SSL certificate from deSEC and a map file to have haproxy proxying different services. For this how-to, the service is nextcloud and the assumption is that haproxy is set as per that Tutorial and there is a service that is both working internally AND is being proxied by haproxy as per that Tutorial. In short, this is an add-on to a service set in that way. The motivation is to add the protection of crowdsec to the protection that haproxy provides.

Next, I have nextcloud installed on a freeBSD jail – type VNET. The front end is running Apache as the webserver. The database is MariaDB, and the nextcloud settings and data are on separate ZFS datasets. That way I can blow the jail up and re-create it, and both data and nextcloud setups remain safe.  This how-to does not require a jail. It can be a virtual machine or another physical host, it doesn't matter.

The final ingredient is a captcha provider. I decided to go with cloudflare turnstile. The setup of it for a new user is described in the crowdsec blog for the haproxy bouncer. The following are the crowdsec materials I used for the overall setup, link3 and link4.
In reality the setup of crowdsec + captcha to haproxy in OPN is a case of adapting file locations from linux to freeBSD and following those two links very closely. Then finding the places to modify the haproxy configuration. That is what we will document here now (finally).

1.    The crowdsec-haproxy-bouncer installation.
Get the latest release from the link provided in link4. Link4 takes you to a crowdsec url, with a link to github. At the time of writing the latest not pre-release is v.0.0.6.

Download the tarball from your working machine and untar it.

Transfer the uncompressed files to your OPNSense. You coud transfer the tarball and extract on OPN of course. Your choice. You will have a directory called lua-mod and four files: "install.sh", "uninstall.sh" and "upgrade.sh". This will be your working directory i.e. /some/path/crowdsec-haproxy-bouncer/crowdsec-haproxy-bouncer-v0.0.6

We can't just run these scripts because a) the locations are not suited for freeBSD (this is fixable);  and b) there is a command "tr" that seems to not be portable to freeBSD and fails (I tried). So instead we'll do manually what the install script is meant to do. You need elevated permissions. Sudo or get root.

Checks:
–    The path /usr/local/lib/crowdsec/ exists.
–    The path /usr/local/etc/crowdsec/ exists
–    The path /var/lib/crowdsec/ exists.

If any of these paths do not exist, then you don't have crowdsec installed or is installed in a non-default location. Find your paths.
·    Verify you are in /some/path/crowdsec-haproxy-bouncer/crowdsec-haproxy-bouncer-v0.0.6

·    Add the bouncer to crowdsec:

Code Select Expand

#sudo cscli bouncers add crowdsec-haproxy-bouncer-AAA
We're adding the AAA suffix so we can identify it. Make a note of the key given by the above command. This is important, it only appears once. If you lose it, you'll need to remove it and create a new one.

·    Add the API key we created to the file crowdsec-haproxy-bouncer.conf inside lua-mod directory, at the top, see snippet:

Code Select Expand

ENABLED=true
API_KEY=${API_KEY}  <== replace from $ to } with the key.
# haproxy
# path to community_blocklist.map
·    Create the subdirectories for the necessary files from the download and copy the files into them:
o   

Code Select Expand

mkdir -p /usr/local/lib/crowdsec/lua/haproxy/plugins/crowdsec/o   

Code Select Expand

mkdir -p /var/lib/crowdsec/lua/haproxy/templates/o   

Code Select Expand

cp -r ./lua-mod/lib/* /usr/local/lib/crowdsec/lua/haproxy/o   

Code Select Expand

cp -r ./lua-mod/templates/* /var/lib/crowdsec/lua/haproxy//templates/o   

Code Select Expand

cp ./lua-mod/community_blocklist.map /var/lib/crowdsec/lua/haproxy/o   

Code Select Expand

cp ./lua-mod/crowdsec-haproxy-bouncer.conf /usr/local/etc/crowdsec/bouncers/

·    Do not restart crowdsec yet. We need to obtain the captcha/turnstile settings required to complete the configuration, so we'll come back to the crowdsec-haproxy-bouncer.conf after step 2.

2. We go to cloudflare's turnstile link5  and sign up to it unless you are already a user. It is free and the traffic doesn't have to go through cloudflare. Follow the link there to "get started" and get your SITEKEY and SECRET KEY.

3. Now go back to the crowdsec-haproxy-bouncer.conf  file and enter there those two values in their respective lines.
We are now ready to finish this configuration file. As well as the two values above, we need to enter the paths for MAP_PATH, BAN_TEMPLATE_PATH and CAPTCHA_TEMPLATE_PATH. It will look like this:

Code Select Expand

ENABLED=true
API_KEY=EJ1HXdgoogoodieUYnR8fzMnSsBVwBu+5uag/bcYA
# haproxy
# path to community_blocklist.map
MAP_PATH=/var/lib/crowdsec/lua/haproxy/community_blocklist.map
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=stream
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
# path to ban template
BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/haproxy/templates/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
# Captcha Secret Key
SECRET_KEY=0x4AACCBBRTRAxxiRDsomc4R89jg8JCfIjOD3g
# captcha Site key
SITE_KEY=0x4ABBTYAAxxiQYQL9NK8XSf
# path to captcha template
CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/haproxy/templates/captcha.html
CAPTCHA_EXPIRATION=3600
The variables without values are not necessary.

4. The haproxy configuration. This one is the one that could cause some trouble if you have a modified setup, especially if we have used the "advanced mode" in the UI sections, because we are going also to use that and pass through some options. If these are also in use by the UI, then we need to find a way to combine them. This is what I had to do a few times until I could find the right place for them. The best way I found to figure things out was to make a change, see the changes in the back end using the "Config Export" option and "Test Syntax".
If your setup is straight as is per the Tutorial, then the following will work.
Note: it is useful to compare it with link4, where this is coming from. Pay attention to lines, they are continuous lines without break, but the formatting wraps them to fit in the boxes.

4.1 Go to HAProxy > Settings. From the Settings tab, use the dropdown to show "Global Parameters", then use the "advanced mode". Now you can add in the field box "Custom options" the following:

Code Select Expand

lua-prepend-path /usr/local/lib/crowdsec/lua/haproxy/?.lua
lua-load /usr/local/lib/crowdsec/lua/haproxy/crowdsec.lua
setenv CROWDSEC_CONFIG /usr/local/etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf

It will look like this:

You cannot view this attachment.

4.2 Go to your front end you want to protect. In the case of the setup we are following, we are going to look to modify "1_HTTPS_frontend" if you named it as such. In my tests I could use a front end in http or https mode but I did not try a TCP mode to apply this. I decided to apply it to the main HTTPS one, as that is the one I want to protect.

So we go to Virtual Services and edit that front end. We need to still be in "advanced mode" and in "Options pass-through" we enter:

Code Select Expand

stick-table type ip size 10k expire 30m # declare a stick table to cache captcha verifications
http-request lua.crowdsec_allow # action to identify crowdsec remediation
http-request track-sc0 src if { var(req.remediation) -m str "captcha-allow" } # cache captcha allow decision
http-request redirect location %[var(req.redirect_uri)] if { var(req.remediation) -m str "captcha-allow" } # redirect to initial url
http-request use-service lua.reply_captcha if { var(req.remediation) -m str "captcha" } # serve captcha template if remediation is captcha
http-request use-service lua.reply_ban if { var(req.remediation) -m str "ban" } # serve ban template if remediation is ban

These are 6 lines that wrap around. To avoid problems with copy and paste, do the copy and paste line by line including the second part that wraps. In the UI, it should scroll along as a single line. In other words for instance the fourth one:

Code Select Expand

http-request redirect location %[var(req.redirect_uri)] if { var(req.remediation) -m str "captcha-allow" } # redirect to initial urlShould not be wrapping and instead finish as a single line in the box.

4.3 Creating a first new server and back end. The server MUST be called turnstile_verifier and the fqdn is challenges.cloudflare.com and the port is 443. Leave the rest with defaults

You cannot view this attachment.

Now create the first back end. It MUST be called captcha_verifier. The server is the "turnstile_verifier" created above. Mode HTTP(Layer7) of course. The rest are defaults.

You cannot view this attachment.

Next post will continue from here due to reaching the allowed size for pictures.

[cookiemonster]

Continuation 1

4.4 Creating a second new server and  back end.
Same principles as the previous pair. The server MUST be called crowdsec and the IP address is the one of your LAPI. In our case the LAPI is on the Settings tab of your crowdsec and the port you have set. Mine is on the LAN ip and on port 8081:

You cannot view this attachment.

Therefore my second new server in haproxy will have those details:

You cannot view this attachment.

Create your second new back end, MUST be called crowdsec, and use the server created above for it.

You cannot view this attachment.

4.5 Now you can restart crowdsec and haproxy. Verify that crowdsec starts successfully. The log is /var/log/crowdsec/crowdsec.log . Haproxy will fail to go into running state from a restart using the UI. You can go in OPN UI to System > Diagnostics > Services to restart haproxy.

Any failure to restart, go over the steps so far. Then we continue.

5. Captcha remediations.
This step is to let crowdsec bouncer know if it should return a captcha or block the ip address coming in based on context. The main purpose of this logic is to deter bots or scripts trying to hammer your public site to guess passwords and such attempts to breach your security. This logic is implemented in profiles.
We go to /usr/local/etc/crowdsec/profiles.yaml and before the default profile:

Code Select Expand

name: default_ip_remediation
#debug: true
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
 - type: ban
   duration: 4h
on_success: breakWe will add our new profile:

Code Select Expand

### Beginning of captcha modification ####
name: captcha_remediation
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Ip" && Alert.GetScenario() startsWith "crowdsecurity/http-"
decisions:
 - type: captcha
   duration: 4h
on_success: break
---
#### End of captcha modification ####
You will notice that the new profile will be looking for alerts that begin with crowdsecurity/http-" and making a decision with a captcha for 4 hours but only for the ip address that triggered the scenario. Now we need to add those scenarios if you don't have them installed, which would me the case in most default installs unless you hare already monitoring for similar ones for another remediation.

Continues

[cookiemonster]

Continuation 2

6. Install base-http-scenarios
As we mentioned just now, crowdsec needs to know how to spot attacks on http services. The base-http-scenarios is a collection of them, see the crowdsec site link6 for details.

Code Select Expand

#cscli collections install crowdsecurity/base-http-scenariosThis will install a the crowdsecurity/base-http-scenarios in the plugin list of Collections, and a bunch scenarios in the Scenarios list. Neat.

You cannot view this attachment.
You cannot view this attachment.

Continues

[cookiemonster]

Continuation 3

7. The final step is to restart both crowdsec and haproxy and you should get this protection at this point. Make sure your services have restarted successfully and revise each step in case of failure to restart.

If your site or service is a quiet one, chances are that a long time will pass before an alert is triggered. In my case I look from time to time as in weeks before I see something new. Screenshot here of what they look like:

You cannot view this attachment.

The manual one you see there was for testing.

Final notes:
- I have not yet found a way to silence the logs in haproxy every 10 seconds, where it checks for decisions to apply. They look like this:

Code Select Expand

2024-12-23T17:46:31    Informational    haproxy    -:- [23/Dec/2024:17:46:31.222] <HTTPCLIENT> -/- 2/0/0/49/49 200 153 - - ---- 0/0/0/0/0 0/0 {} "GET http://192.168.5.1:8081/v1/decisions/stream?startup=false HTTP/1.1"

[orxcyd]

thank you for write this guide, i tried and seem to work.

i think you forgot include "cp ./lua-mod/crowdsec-haproxy-bouncer.conf /usr/local/etc/crowdsec/bouncers/"

[cookiemonster]

thank you for write this guide, i tried and seem to work.

i think you forgot include "cp ./lua-mod/crowdsec-haproxy-bouncer.conf /usr/local/etc/crowdsec/bouncers/"

Thank you for trying it out and providing the feedback, much appreciated. Some of it was from memory.
I think the original install script "sudo cscli bouncers add crowdsec-haproxy-bouncer-${SUFFIX} -o raw" would have added the config file and after it failed on me, I didn't check what was left in place. I will add the copy to the instructions. Thanks.