I'm running OPNsense 24.7 with HAProxy and Crowdsec. I have several services running behind HAProxy some of them with Crowdsec log parsers installed, reporting to the OPNsense Crowdsec LAPI. The firewall bouncer works great with this setup, but I also want to block Traffic at Layer 7 directly on HAProxy. This is where the Crowdsec HAProxy Bouncer comes into play, but I can't get it running on OPNsense. Is there any chance to get the bouncer working?
https://github.com/crowdsecurity/cs-haproxy-bouncer
https://www.crowdsec.net/blog/the-haproxy-bouncer-is-out
https://docs.crowdsec.net/u/bouncers/haproxy/
Second this for the nginx bouncer as well. I'm currently considering moving nginx off of my opnsense box just to have this functionality but would ideally like to keep it where it's at. Handosense was trying to get the HaProxy bouncer working too - https://forum.opnsense.org/index.php?topic=39377.0 (https://forum.opnsense.org/index.php?topic=39377.0)
Its probably easier to get the caddy crowdsec bouncer running with the os-caddy plugin, since it can be compiled into the caddy binary and then you can just configure it in the caddyfile.
There are no outside dependencies.
Adding that would be very straight forward, so if somebody offers a PR I would review it.
https://caddyserver.com/docs/modules/crowdsec
What use is an HAproxy/Caddy/NginX bouncer if the firewall bouncer kicked in already? Attackers will never be able to contact your L 7 services to get bounced.
Well you are right that it does not seem to make sense.
I just feed the logs to OPNsense and it blocks via pf.
For me, I use Cloudflare proxy and my WAN only accepts inbound from the Cloudflare IP ranges (https://www.cloudflare.com/ips/ (https://www.cloudflare.com/ips/). That's all the firewall sees so it can't block by the remote IP that nginx sees. I could set up the bouncer on each one of my services but would prefer to have it right on the reverse proxy.
I have been wanting also the that for haproxy New/unknown IP are checked against crowdsec API and have been disappointed that the bouncer is not available for OPN/freeBSD.
I could very well be mistaken but my reasoning is that haproxy has ports exposed via firewall allow rules, so the firewall has handed over the connection to haproxy. Here is where crowdsec would come to do its thing.
Is that incorrect reasoning ?
Quote from: JLDC on October 12, 2024, 04:50:01 PM
For me, I use Cloudflare proxy and my WAN only accepts inbound from the Cloudflare IP ranges (https://www.cloudflare.com/ips/ (https://www.cloudflare.com/ips/). That's all the firewall sees so it can't block by the remote IP that nginx sees. I could set up the bouncer on each one of my services but would prefer to have it right on the reverse proxy.
Now the usecase is taking shape. The opnsense wont see the proxy protocol or x-forwarded-for header and the block on opnsense firewall level would be useless.
Like written above, trivial to add crowdsec to the existing caddy plugin.
@Monviech would you be OK to share your thoughts on my question please? I would appreciate your input.
If you dont mask the IPs of the incoming requests with a CDN you can just use the crowdsec opnsense bouncer, let it parse the ha proxy http access logs and block on firewall level.
I do the same parsing the caddy logs. It works. The CDN usecase is special.
No CDN in front of haproxy so it sounds like it is doable. I'll see if I can figure out how to implement what you suggest. Thanks.
Here is how it currently works in caddy with the http access logs for reference.
Its really simple:
https://docs.opnsense.org/manual/how-tos/caddy.html#crowdsec-integration
(thanks @Patrick here who suggested this approach at the beginning :)
Quote from: Monviech on October 15, 2024, 12:22:28 PM
Here is how it currently works in caddy with the http access logs for reference.
Its really simple:
https://docs.opnsense.org/manual/how-tos/caddy.html#crowdsec-integration
(thanks @Patrick here who suggested this approach at the beginning :)
Many thanks. I didn't even know the docs has had a facelift.
An update on this one. I did follow that suggestion, modified the crowdsec opnsense bouncer but perhaps because the logging in haproxy is so different, it seemed that it had no way to trigger an action.
So I went to see if I could figure out what the haproxy bouncer gets installed and does its thing.
I have been able to install it and configure it and so far both it shows running and I have finished clearing out the last setup failure on haproxy.
An internal test seems to suggest that the action was correctly triggered (in my case I setup a captcha, in reality I signed up to cloudflare turnstile just for this).
I am however not been able to confirm it works for external entities trying to abuse so as to trigger it. Once I've been able to do that and I am more sure that is doing what it is supposed to, I shall be posting a description of what I did.
Update. It seems to be working correctly and serving both bans and captchas.
One last thing to diagnose is a loop I get on the captcha or more specifically the bot-catching "captcha" aka turnstile. Almost there I think.
I've asked crowdsec in various places now just awaiting input.
Quote from: Monviech (Cedrik) on October 12, 2024, 05:25:31 PMQuote from: JLDC on October 12, 2024, 04:50:01 PMFor me, I use Cloudflare proxy and my WAN only accepts inbound from the Cloudflare IP ranges (https://www.cloudflare.com/ips/ (https://www.cloudflare.com/ips/). That's all the firewall sees so it can't block by the remote IP that nginx sees. I could set up the bouncer on each one of my services but would prefer to have it right on the reverse proxy.
Now the usecase is taking shape. The opnsense wont see the proxy protocol or x-forwarded-for header and the block on opnsense firewall level would be useless.
Like written above, trivial to add crowdsec to the existing caddy plugin.
This would also be my use case for Cloudflare. You can pass the X-Forwarded-For header from Cloudflare's CF-Connecting-IP. For this to work, the bounce has to happen at Layer 7.
--
Also, the log parser for HAProxy, which should trigger the firewall bouncer, doesn't seem to work because the log files are not in the standard format on OPNsense, I guess.
Current metrics:
- (Parser) crowdsecurity/haproxy-logs:
╭──────────────────────────────────┬──────┬────────┬──────────╮
│ Parsers │ Hits │ Parsed │ Unparsed │
├──────────────────────────────────┼──────┼────────┼──────────┤
│ file:/var/log/haproxy/latest.log │ 115 │ 0 │ 115 │
╰──────────────────────────────────┴──────┴────────┴──────────╯
Quote from: cookiemonster on October 21, 2024, 11:09:25 PMUpdate. It seems to be working correctly and serving both bans and captchas.
One last thing to diagnose is a loop I get on the captcha or more specifically the bot-catching "captcha" aka turnstile. Almost there I think.
I've asked crowdsec in various places now just awaiting input.
Did you get it to work?
Maybe we can also Ping this guy if you found the solution :)
https://forum.opnsense.org/index.php?topic=39377.0
Quote from: cookiemonster on October 21, 2024, 11:09:25 PMUpdate. It seems to be working correctly and serving both bans and captchas.
One last thing to diagnose is a loop I get on the captcha or more specifically the bot-catching "captcha" aka turnstile. Almost there I think.
I've asked crowdsec in various places now just awaiting input.
Did you get it to work?
Yes I did and made a small How-To here https://forum.opnsense.org/index.php?topic=44839.0
And you are correct and my suspicion also was, the same thinking. There is no use in giving the HA logs to Crowdsec for that very reason. It is not aware how to process them. With my how-to, we make it it aware.
Quote from: cookiemonster on October 21, 2024, 11:09:25 PMYes I did and made a small How-To here https://forum.opnsense.org/index.php?topic=44839.0
And you are correct and my suspicion also was, the same thinking. There is no use in giving the HA logs to Crowdsec for that very reason. It is not aware how to process them. With my how-to, we make it it aware.
This is great!
Thank you very much for your effort.
btw.
You can use this conditions and rules to forward the X-forwarded-for header to the application if you're behind a Cloudflare Proxy. That way the Application sees the Real Client ip and can make decisions based on them.
acl1 src -f /var/haproxy/CF_ips.lst
acl2 req.hdr(CF-Connecting-IP) -m found
# ACTION: Cloudflare_real_ip
http-request set-header X-Forwarded-For %[req.hdr(CF-Connecting-IP)] if acl1 acl2
# ACTION: cf_set_scr
http-request set-src hdr(x-forwarded-for) if acl1
# ACTION: cf_set_var
http-request set-var(txn.sess.cloudflare) always_true if acl1 Hit me up if you need more details
Thanks for this input @Zenturio. I am not behind a cloudflare proxy but some others might be. Good addition of information.