Crowdsec and external servers/services

[Matzke]

Dear all,
I'm pretty new to crowdsec and have a little question.

I setup crowdsec on opnsense and I think it works as it should.

Now I have some internal servers which are connected via port-forwarding in opnsense to the internet (either port-forwarding or HAProxy reverse proxy on opnsense).

What do I need to collect and analyze the log files of these dedicated servers and report it to crowdsec running on my opnsense? The blocking-rules should also happens on opnsense because this is the gatekeeper to the internet.

Are there any manuals out there for this scenario?

Thanks a lot

[iMx]

Port Forwarding
Not much you can do here, as there are no logs as such to parse. 

However, blocked IPs will also be blocked from port forwards, due to the floating drop rule.

Code: [Select]

pfctl -s rules | grep crowd
block drop in quick inet from <crowdsec_blacklists> to any label "6fc904ee8f33bb90e1c73147d55cd852"
block drop in quick inet6 from <crowdsec6_blacklists> to any label "7de971956cb806447b5f10bdb3d4d9bb"
HAproxy
For HAProxy you can install the haproxy collections:

Code: [Select]

cscli collections install crowdsecurity/haproxy
Then in:

/usr/local/etc/crowdsec/acquis.d

Create haproxy.yaml:

Code: [Select]

filenames:
 - /var/log/haproxy/latest.log
force_inotify: true
poll_without_inotify: true
labels:
  type: syslog
.. then reload crowdsec and check you see it in the log file:

Code: [Select]

grep hap /var/log/crowdsec/crowdsec.log
and in metrics:

Code: [Select]

cscli metrics | grep haproxy
Distributed setup

If there are parsers available for the backend services - i.e the port forward destination, or the haproxy proxy real/backend servers - you can install Crowdsec on them, then add them as 'machines' in Crowdsec on opnsense so they report back to opnsense.

You would need to change the LAPI from 127.0.0.1 so its network accessible.

... for example, I have:

- HAproxy running on opnsense, for ssl/frontend for HomeAssistant
- I installed the HAProxy collection, in Crowdsec on opnsense
- Crowdsec will then parse the HAproxy logs
- Then Crowdsec installed on HomeAssistant parsing its logs
- HomeAssistant then reports back to opnsense (as the bouncer/blocker) and adds block rules there

This means opnsense can parse HAproxy logs, for what little information is available there.  Then Crowdsec on HomeAssistant parses actual access logs and reports back to opnsense.

[iMx]

Forgot to add....

For Haproxy log parsing you need to enable 'Detailed Logging' so that the parser will match:

- Services
- Haproxy
- Settings
- Virtual Services -> Public Services
- Edit your frontends
- Enable 'Detailed Logging'

Logs look like they should then match the parser:

s01-parse/haproxy-logs.yaml

Code: [Select]

  pattern: '%{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"'
N.B. Assume HAProxy 'Type' really needs to be HTTP/HTTPS - as the parser looks for the HTTP response codes - can't really do a lot (anything?) with Layer4/TCP (perhaps basic DoS prevention, queue, etc)

[Patrick M. Hausen]

For parsing application specific logs you need to install log collectors on the servers themselves and connect them to your crowdsec engine. How that is done depends on the server OS and your preferred method of deployment, of course.

See screenshot for what I am running.