"
I have been meaning to write this small how-to for a little while. It is working for me and I will not pretend to be fully understanding of Crowdsec's ins and outs. It wouldn't surprise me if there are incorrect assumptions or errors in my understanding. No warranties given but with that out of the way, first some context.
I have a handful of services that I expose to the open internet but only for me to use. These services are hosted on my LAN. I use layers of security and is unusual for me to use simple port forwards. Instead, services are usually behind a reverse proxy (haproxy) which sits on OPNSense, plus the usual additional protections like fail2ban and other methods.
This how-to is to add crowdsec captcha protection to haproxy on OPNSense, specifically to the haproxy plugin. It might have been possible to get crowdsec to read the httpd-access and httpd-error logs from Apache on the freeBSD jail it is protecting but I wanted to separate the tests I was doing at the time, creating and destroying jails frequently. With my preferred setup, crowdsec is protecting all services behind all jails in a central place. Additionally I don't have to setup crowdsec on each target jail, virtual machine or host accessed via haproxy.
The simplified logical setup is as follows:
You cannot view this attachment.
I already have a crowdsec distributed setup link1, where the LAPI runs on OPN, using the crowdsec plugin. The plugin installs the default crowdsec firewall bouncer and other components that are part of the crowdsec/opnsense collection. For this how-to, it is assumed that crowdsec has been setup on OPN already and is working with the LAPI enabled. There is no need for a distributed setup but it will work just fine in a distributed or standalone setup. Note 1: for clarity, the link to the distributed setup is one I used to base my setup but I did it differently to the example where my setup is reversed from the example i.e. the public machine has the LAPI.
I have haproxy on OPN setup as per the @thehellsite's guide link2. I have a public wildcard SSL certificate from deSEC and a map file to have haproxy proxying different services. For this how-to, the service is nextcloud and the assumption is that haproxy is set as per that Tutorial and there is a service that is both working internally AND is being proxied by haproxy as per that Tutorial. In short, this is an add-on to a service set in that way. The motivation is to add the protection of crowdsec to the protection that haproxy provides.
Next, I have nextcloud installed on a freeBSD jail – type VNET. The front end is running Apache as the webserver. The database is MariaDB, and the nextcloud settings and data are on separate ZFS datasets. That way I can blow the jail up and re-create it, and both data and nextcloud setups remain safe. This how-to does not require a jail. It can be a virtual machine or another physical host, it doesn't matter.
The final ingredient is a captcha provider. I decided to go with cloudflare turnstile. The setup of it for a new user is described in the crowdsec blog for the haproxy bouncer. The following are the crowdsec materials I used for the overall setup, link3 and link4.
In reality the setup of crowdsec + captcha to haproxy in OPN is a case of adapting file locations from linux to freeBSD and following those two links very closely. Then finding the places to modify the haproxy configuration. That is what we will document here now (finally).
1. The crowdsec-haproxy-bouncer installation.
Get the latest release from the link provided in link4. Link4 takes you to a crowdsec url, with a link to github. At the time of writing the latest not pre-release is v.0.0.6.
Download the tarball from your working machine and untar it.
Transfer the uncompressed files to your OPNSense. You coud transfer the tarball and extract on OPN of course. Your choice. You will have a directory called lua-mod and four files: "install.sh", "uninstall.sh" and "upgrade.sh". This will be your working directory i.e. /some/path/crowdsec-haproxy-bouncer/crowdsec-haproxy-bouncer-v0.0.6
We can't just run these scripts because a) the locations are not suited for freeBSD (this is fixable); and b) there is a command "tr" that seems to not be portable to freeBSD and fails (I tried). So instead we'll do manually what the install script is meant to do. You need elevated permissions. Sudo or get root.
Checks:
– The path /usr/local/lib/crowdsec/ exists.
– The path /usr/local/etc/crowdsec/ exists
– The path /var/lib/crowdsec/ exists.
If any of these paths do not exist, then you don't have crowdsec installed or is installed in a non-default location. Find your paths.
· Verify you are in /some/path/crowdsec-haproxy-bouncer/crowdsec-haproxy-bouncer-v0.0.6
· Add the bouncer to crowdsec:
We're adding the AAA suffix so we can identify it. Make a note of the key given by the above command. This is important, it only appears once. If you lose it, you'll need to remove it and create a new one.
· Add the API key we created to the file crowdsec-haproxy-bouncer.conf inside lua-mod directory, at the top, see snippet:
· Create the subdirectories for the necessary files from the download and copy the files into them:
o
· Do not restart crowdsec yet. We need to obtain the captcha/turnstile settings required to complete the configuration, so we'll come back to the crowdsec-haproxy-bouncer.conf after step 2.
2. We go to cloudflare's turnstile link5 and sign up to it unless you are already a user. It is free and the traffic doesn't have to go through cloudflare. Follow the link there to "get started" and get your SITEKEY and SECRET KEY.
3. Now go back to the crowdsec-haproxy-bouncer.conf file and enter there those two values in their respective lines.
We are now ready to finish this configuration file. As well as the two values above, we need to enter the paths for MAP_PATH, BAN_TEMPLATE_PATH and CAPTCHA_TEMPLATE_PATH. It will look like this:
The variables without values are not necessary.
4. The haproxy configuration. This one is the one that could cause some trouble if you have a modified setup, especially if we have used the "advanced mode" in the UI sections, because we are going also to use that and pass through some options. If these are also in use by the UI, then we need to find a way to combine them. This is what I had to do a few times until I could find the right place for them. The best way I found to figure things out was to make a change, see the changes in the back end using the "Config Export" option and "Test Syntax".
If your setup is straight as is per the Tutorial, then the following will work.
Note: it is useful to compare it with link4, where this is coming from. Pay attention to lines, they are continuous lines without break, but the formatting wraps them to fit in the boxes.
4.1 Go to HAProxy > Settings. From the Settings tab, use the dropdown to show "Global Parameters", then use the "advanced mode". Now you can add in the field box "Custom options" the following:
It will look like this:
You cannot view this attachment.
4.2 Go to your front end you want to protect. In the case of the setup we are following, we are going to look to modify "1_HTTPS_frontend" if you named it as such. In my tests I could use a front end in http or https mode but I did not try a TCP mode to apply this. I decided to apply it to the main HTTPS one, as that is the one I want to protect.
So we go to Virtual Services and edit that front end. We need to still be in "advanced mode" and in "Options pass-through" we enter:
These are 6 lines that wrap around. To avoid problems with copy and paste, do the copy and paste line by line including the second part that wraps. In the UI, it should scroll along as a single line. In other words for instance the fourth one:
4.3 Creating a first new server and back end. The server MUST be called turnstile_verifier and the fqdn is challenges.cloudflare.com and the port is 443. Leave the rest with defaults
You cannot view this attachment.
Now create the first back end. It MUST be called captcha_verifier. The server is the "turnstile_verifier" created above. Mode HTTP(Layer7) of course. The rest are defaults.
You cannot view this attachment.
I have a handful of services that I expose to the open internet but only for me to use. These services are hosted on my LAN. I use layers of security and is unusual for me to use simple port forwards. Instead, services are usually behind a reverse proxy (haproxy) which sits on OPNSense, plus the usual additional protections like fail2ban and other methods.
This how-to is to add crowdsec captcha protection to haproxy on OPNSense, specifically to the haproxy plugin. It might have been possible to get crowdsec to read the httpd-access and httpd-error logs from Apache on the freeBSD jail it is protecting but I wanted to separate the tests I was doing at the time, creating and destroying jails frequently. With my preferred setup, crowdsec is protecting all services behind all jails in a central place. Additionally I don't have to setup crowdsec on each target jail, virtual machine or host accessed via haproxy.
The simplified logical setup is as follows:
You cannot view this attachment.
I already have a crowdsec distributed setup link1, where the LAPI runs on OPN, using the crowdsec plugin. The plugin installs the default crowdsec firewall bouncer and other components that are part of the crowdsec/opnsense collection. For this how-to, it is assumed that crowdsec has been setup on OPN already and is working with the LAPI enabled. There is no need for a distributed setup but it will work just fine in a distributed or standalone setup. Note 1: for clarity, the link to the distributed setup is one I used to base my setup but I did it differently to the example where my setup is reversed from the example i.e. the public machine has the LAPI.
I have haproxy on OPN setup as per the @thehellsite's guide link2. I have a public wildcard SSL certificate from deSEC and a map file to have haproxy proxying different services. For this how-to, the service is nextcloud and the assumption is that haproxy is set as per that Tutorial and there is a service that is both working internally AND is being proxied by haproxy as per that Tutorial. In short, this is an add-on to a service set in that way. The motivation is to add the protection of crowdsec to the protection that haproxy provides.
Next, I have nextcloud installed on a freeBSD jail – type VNET. The front end is running Apache as the webserver. The database is MariaDB, and the nextcloud settings and data are on separate ZFS datasets. That way I can blow the jail up and re-create it, and both data and nextcloud setups remain safe. This how-to does not require a jail. It can be a virtual machine or another physical host, it doesn't matter.
The final ingredient is a captcha provider. I decided to go with cloudflare turnstile. The setup of it for a new user is described in the crowdsec blog for the haproxy bouncer. The following are the crowdsec materials I used for the overall setup, link3 and link4.
In reality the setup of crowdsec + captcha to haproxy in OPN is a case of adapting file locations from linux to freeBSD and following those two links very closely. Then finding the places to modify the haproxy configuration. That is what we will document here now (finally).
1. The crowdsec-haproxy-bouncer installation.
Get the latest release from the link provided in link4. Link4 takes you to a crowdsec url, with a link to github. At the time of writing the latest not pre-release is v.0.0.6.
Download the tarball from your working machine and untar it.
Transfer the uncompressed files to your OPNSense. You coud transfer the tarball and extract on OPN of course. Your choice. You will have a directory called lua-mod and four files: "install.sh", "uninstall.sh" and "upgrade.sh". This will be your working directory i.e. /some/path/crowdsec-haproxy-bouncer/crowdsec-haproxy-bouncer-v0.0.6
We can't just run these scripts because a) the locations are not suited for freeBSD (this is fixable); and b) there is a command "tr" that seems to not be portable to freeBSD and fails (I tried). So instead we'll do manually what the install script is meant to do. You need elevated permissions. Sudo or get root.
Checks:
– The path /usr/local/lib/crowdsec/ exists.
– The path /usr/local/etc/crowdsec/ exists
– The path /var/lib/crowdsec/ exists.
If any of these paths do not exist, then you don't have crowdsec installed or is installed in a non-default location. Find your paths.
· Verify you are in /some/path/crowdsec-haproxy-bouncer/crowdsec-haproxy-bouncer-v0.0.6
· Add the bouncer to crowdsec:
Code Select
#sudo cscli bouncers add crowdsec-haproxy-bouncer-AAAWe're adding the AAA suffix so we can identify it. Make a note of the key given by the above command. This is important, it only appears once. If you lose it, you'll need to remove it and create a new one.
· Add the API key we created to the file crowdsec-haproxy-bouncer.conf inside lua-mod directory, at the top, see snippet:
Code Select
ENABLED=true
API_KEY=${API_KEY} <== replace from $ to } with the key.
# haproxy
# path to community_blocklist.map· Create the subdirectories for the necessary files from the download and copy the files into them:
o
Code Select
mkdir -p /usr/local/lib/crowdsec/lua/haproxy/plugins/crowdsec/o Code Select
mkdir -p /var/lib/crowdsec/lua/haproxy/templates/o Code Select
cp -r ./lua-mod/lib/* /usr/local/lib/crowdsec/lua/haproxy/o Code Select
cp -r ./lua-mod/templates/* /var/lib/crowdsec/lua/haproxy//templates/o Code Select
cp ./lua-mod/community_blocklist.map /var/lib/crowdsec/lua/haproxy/o Code Select
cp ./lua-mod/crowdsec-haproxy-bouncer.conf /usr/local/etc/crowdsec/bouncers/· Do not restart crowdsec yet. We need to obtain the captcha/turnstile settings required to complete the configuration, so we'll come back to the crowdsec-haproxy-bouncer.conf after step 2.
2. We go to cloudflare's turnstile link5 and sign up to it unless you are already a user. It is free and the traffic doesn't have to go through cloudflare. Follow the link there to "get started" and get your SITEKEY and SECRET KEY.
3. Now go back to the crowdsec-haproxy-bouncer.conf file and enter there those two values in their respective lines.
We are now ready to finish this configuration file. As well as the two values above, we need to enter the paths for MAP_PATH, BAN_TEMPLATE_PATH and CAPTCHA_TEMPLATE_PATH. It will look like this:
Code Select
ENABLED=true
API_KEY=EJ1HXdgoogoodieUYnR8fzMnSsBVwBu+5uag/bcYA
# haproxy
# path to community_blocklist.map
MAP_PATH=/var/lib/crowdsec/lua/haproxy/community_blocklist.map
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=stream
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
# path to ban template
BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/haproxy/templates/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
# Captcha Secret Key
SECRET_KEY=0x4AACCBBRTRAxxiRDsomc4R89jg8JCfIjOD3g
# captcha Site key
SITE_KEY=0x4ABBTYAAxxiQYQL9NK8XSf
# path to captcha template
CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/haproxy/templates/captcha.html
CAPTCHA_EXPIRATION=3600The variables without values are not necessary.
4. The haproxy configuration. This one is the one that could cause some trouble if you have a modified setup, especially if we have used the "advanced mode" in the UI sections, because we are going also to use that and pass through some options. If these are also in use by the UI, then we need to find a way to combine them. This is what I had to do a few times until I could find the right place for them. The best way I found to figure things out was to make a change, see the changes in the back end using the "Config Export" option and "Test Syntax".
If your setup is straight as is per the Tutorial, then the following will work.
Note: it is useful to compare it with link4, where this is coming from. Pay attention to lines, they are continuous lines without break, but the formatting wraps them to fit in the boxes.
4.1 Go to HAProxy > Settings. From the Settings tab, use the dropdown to show "Global Parameters", then use the "advanced mode". Now you can add in the field box "Custom options" the following:
Code Select
lua-prepend-path /usr/local/lib/crowdsec/lua/haproxy/?.lua
lua-load /usr/local/lib/crowdsec/lua/haproxy/crowdsec.lua
setenv CROWDSEC_CONFIG /usr/local/etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf
It will look like this:
You cannot view this attachment.
4.2 Go to your front end you want to protect. In the case of the setup we are following, we are going to look to modify "1_HTTPS_frontend" if you named it as such. In my tests I could use a front end in http or https mode but I did not try a TCP mode to apply this. I decided to apply it to the main HTTPS one, as that is the one I want to protect.
So we go to Virtual Services and edit that front end. We need to still be in "advanced mode" and in "Options pass-through" we enter:
Code Select
stick-table type ip size 10k expire 30m # declare a stick table to cache captcha verifications
http-request lua.crowdsec_allow # action to identify crowdsec remediation
http-request track-sc0 src if { var(req.remediation) -m str "captcha-allow" } # cache captcha allow decision
http-request redirect location %[var(req.redirect_uri)] if { var(req.remediation) -m str "captcha-allow" } # redirect to initial url
http-request use-service lua.reply_captcha if { var(req.remediation) -m str "captcha" } # serve captcha template if remediation is captcha
http-request use-service lua.reply_ban if { var(req.remediation) -m str "ban" } # serve ban template if remediation is banThese are 6 lines that wrap around. To avoid problems with copy and paste, do the copy and paste line by line including the second part that wraps. In the UI, it should scroll along as a single line. In other words for instance the fourth one:
Code Select
http-request redirect location %[var(req.redirect_uri)] if { var(req.remediation) -m str "captcha-allow" } # redirect to initial urlShould not be wrapping and instead finish as a single line in the box.4.3 Creating a first new server and back end. The server MUST be called turnstile_verifier and the fqdn is challenges.cloudflare.com and the port is 443. Leave the rest with defaults
You cannot view this attachment.
Now create the first back end. It MUST be called captcha_verifier. The server is the "turnstile_verifier" created above. Mode HTTP(Layer7) of course. The rest are defaults.
You cannot view this attachment.
Next post will continue from here due to reaching the allowed size for pictures.
