Caddy, Cloudflare proxy and Client IP Headers

[Lemimouth]

Hi,

I just set up Caddy with CorwdSec integration by following the doc (https://docs.opnsense.org/manual/how-tos/caddy.html#crowdsec-integration). How can I be sure that CrowdSec is parsing Caddy logs correctly ?

My external domain is proxified by Cloudflare, so in Caddy logs, the real client ip address is not in the remote_ip header, but in the X-Forwarded-For and Cf-Connecting-Ip headers.

I've checked the CrowdSec's Caddy parser configuration here : https://app.crowdsec.net/hub/author/crowdsecurity/log-parsers/caddy-logs

But it's not clear :



Thanks !

[Monviech (Cedrik)]

Did you set these headers in the General Settings of the Caddy plugin?

I think I added it upon request just for these usecases:

https://github.com/opnsense/plugins/issues/4517

[Monviech (Cedrik)]

https://caddyserver.com/docs/json/apps/http/servers/client_ip_headers/#docs

This states it will use it to replace the client_ip

trusted_proxies and client_ip_headers can both be configured from the GUI.

[Lemimouth]

I didn't set these headers earlier because, to be honest, I hadn't figured out how to. But now it's done! 😉 (For the record, I had to add them first in Reverse Proxy > Headers, and only then could I select them in General > Advanced).

Thanks ! step 1 is done.

Regarding the trusted proxies, is there a way to automate this ?

In the Github issue, I can see :

Code Select Expand

trusted_proxies static <ips> # https://www.cloudflare.com/ips/
Do I have to manually add all ranges from https://www.cloudflare.com/ips-v4/# ?

[Monviech (Cedrik)]

Yeah you have to create an access list with all of the ips inside.

Just set it to text mode and copy the block in, one IP per line.

That way its pretty fast configured.

[Lemimouth]

Well the Caddy part seems to work. In Diagnostics > Caddyfile I have now :

Code Select Expand

# Global Options
{
log {
output net unixgram//var/run/caddy/log.sock {
}
format json {
time_format rfc3339
}
}

servers {
protocols h1 h2 h3
trusted_proxies static 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/13 104.24.0.0/14 172.64.0.0/13 131.0.72.0/22
client_ip_headers Cf-Connecting-Ip
client_ip_headers X-Forwarded-For
}

auto_https off
grace_period 10s
import /usr/local/etc/caddy/caddy.d/*.global
}

In Caddy log, the client_ip field shows my real ip address :

Code Select Expand

"request": {
        "remote_ip": "162.158.xxx.xx",
        "remote_port": "48400",
        "client_ip": "194.xxx.xxx.xx",
        "proto": "HTTP/2.0",

Now I have to check if CrowdSed is doing it's job.

When doing cscli metrics, I can see that the counter for crowdsecurity/caddy-logs is increasing after each request, so it's a good sign.

Thank again !

[Monviech (Cedrik)]

Cool, nice confirmation the Crowdsec setup works like this with Cloudflare.

Means no other bouncer plugins are needed, it all works built in.

[Lemimouth]

After doing some more tests, this in fact doesn't work, at least with the default bouncer that is installed with the CrowsSec plugin.

The reason is that in Caddy's log, I have now the correct value for the client_ip field (the real ip of the client), and CrowdSec blocks connections based on this ip. But as my domain is behind Cloudflare, the connection comes from the CF proxy ip address and not directly from the client.

I disabled CF proxy for the moment, and it's working again.

Also, I added the following block in /usr/local/etc/caddy/caddy.d/logger.conf :

Code Select Expand

:443 {
log {
output file /var/log/caddy/access/access.log {
roll_keep_for 10d
}
}

handle {
        abort
    }
}

If not doing that, Caddy is logging access ONLY for the configured domains in Reverse Proxy > Domains.

[Monviech (Cedrik)]

Have you tried to remove X-Forwarded-For from the client_ip headers and only add the CF header?

The Caddy documentation states it takes the best first match if multiple client ip headers are specified, maybe it works to only choose the Cloudflare header and skip x-forwarded-for?

It says here its the real IP:
https://developers.cloudflare.com/fundamentals/reference/http-headers/#cf-connecting-ip

[Monviech (Cedrik)]

Oh wait I misunderstood it. The IPs would be put into the Crowdsec alias, but it wouldnt matter since on layer 3 the IP would be always cloudflare so the firewall rule would not match.

It would need to be added into caddy with one of those bouncer plugins to work.

[Lemimouth]

It would need to be added into caddy with one of those bouncer plugins to work.

Yes exactly.

I'm not sure it's (easily) feasible on OPNSense because, to my knowledge, Caddy must be compiled with the caddy-crowdsec-bouncer plugin. It's not something you can enable just by installing a plugin.

I'll disable CF proxy for now and let CrowdSec doing it's job.

Thanks for you time helping me ;)

[Monviech (Cedrik)]

Thanks for looking into this.

It is feasable to add it but I dont really want to add it since every compiled package more just makes maintanance harder in the long run.

Though there is always "caddy add-package" for the more adventurous :)

[Lemimouth]

I wasn't aware of the add-package command, nice to know ! I also use Caddy in Docker in other parts of my network and simply build it with xcaddy to include the plugins I use. I don't feel THAT adventurous with my OPNSense installation. ;)

You're right, the Caddy's OPNSense plugin should be as generic as possible for ease of maintenance. This is totally not a feature request ;)