Messages

As the CrowdSec default firewall is only stopping incoming for items on the list, I wanted to upgrade how that feature worked, and honestly allow a few hosts I have to not be blocked by the firewall - an unfiltered host if you will.

So, I made my own "Hosts" based Firewall Alias, and have a Python script that will get the latest list and put it in there.

This took a little bit usually, and so I tried to see if I could thread the operation to increase speed, I might try to multi-process it next as the dual sends is about the fastest send (two sub lists, each about 30,000 items) and if there is a change I just update the whole sublist that changed and do a reconfigure.

Still, this takes too long as 10 seconds. The other kinds of aliases are interesting to me, such as Internal and External.

It would seem (and I did this too... but didn't understand how to 'reconfigure' or set the updates as active) that you can do it faster via pfctl via python, but, how do you reconfigure after updating a "Hosts" based Alias? Do the Internal or External Alias types not need a 'reconfigure' to have their populations be active in the rules that use them?

I'm fairly new to pf/FreeBSD so please do not take for granted anything I might 'should' or 'could' know, teach me!

Issue:
Last two updates, on restart, WAN has IPv6 address but no IPv4 address

Further:
The solve is easy, I just go to WAN interface and hit 'Save' at the bottom, it applies, and then I have an IPv4 address on WAN followed by a little later the IPv6 address shows up again on WAN. LAN interfaces have their IPv6 (and IPv4) details, but the missing IPv4 on WAN is initially keeping a few things from working after the update.

I will be able to do more testing later as far as the reboot w/out an update, but I do not have that change management window right now.

While I do not know much about OSPF, I have looked up a lot of tuning elements for OPNSense and FreeBSD as mine runs on older metal host and I'm doing 10G intranet.

https://calomel.org/freebsd_network_tuning.html

This website ^ has details about many tuning elements, but they use a different reference of values for kern.ipc.maxsockbuf and so while I do not think there is a limitation/drawback to increasing the value (much the opposite it seems), using a logical value seems wise.

For my router, I have the value set at: 614400000

Seems yours is already working, but who knows if there is odd grouping/read/writes to that space due to a unqiue value - that said, how this values comes to be seems pretty odd to me, and so I didn't do all the work necessary to evaluate your "33554432" - just wanted to share a resource I found that has helped lower buffer bloat and latency.

The specific part from their website that seems important:
# speed:   1 Gbit   maxsockbuf:   2MB   wscale:  6   in-flight:  2^6*65KB =    4MB (default)
# speed:   2 Gbit   maxsockbuf:   4MB   wscale:  7   in-flight:  2^7*65KB =    8MB
# speed:  10 Gbit   maxsockbuf:  16MB   wscale:  9   in-flight:  2^9*65KB =   32MB
# speed:  40 Gbit   maxsockbuf: 150MB   wscale: 12   in-flight: 2^12*65KB =  260MB
# speed: 100 Gbit   maxsockbuf: 600MB   wscale: 14   in-flight: 2^14*65KB = 1064MB
#
#kern.ipc.maxsockbuf=2097152    # (wscale  6 ; default)
#kern.ipc.maxsockbuf=4194304    # (wscale  7)
kern.ipc.maxsockbuf=16777216   # (wscale  9)
#kern.ipc.maxsockbuf=157286400  # (wscale 12)
#kern.ipc.maxsockbuf=614400000   # (wscale 14)

External hosts includes internal IPs, Internal hosts includes external IPs

I'm only in detection for 3 LAN Interfaces, I do have RSS enabled (it suggests I disable it... but it has been around for years now and certainly appears to work for everything else including Suricata)

The "Traffic Graph (Throughput)" Dashboard that shows activity, only shows activity for Upload, not Download (even if I speed test Up/Down, just shows Upload)

Any ideas, I'm completely new to Zenarmor and have a free account but have it integrated to their Cloud too?

That said, I have had a few detections for hosts going to odd FQDNs, not bad!

There's a patch:

Code Select Expand

opnsense-patch -c plugins a80156815
should've read the posts before posting... thank you everyone and Franco!!!

I have several WOL hosts on my network, and I can trigger them to turn on, that works.

What does not work is the 'active status' on the WOL Plug-in Dashboard on the OPNSense. Further, it seems you have to 'wake all' now from the Wake On LAN plug-in menu... it seems you cannot select a specific device to wake up when in the plug-ins menu?

As the CrowdSec Parser Agent that is installed will parse what it is told to from the `/usr/local/etc/crowdsec/acquis.d/*.yaml` and `/usr/local/etc/crowdsec/acquis.yaml` on the OPNSense, it is more about the detail there, and the Allowlists and other pre and post processing you configure.

That all said, by default the plug-in's CrowdSec Agent Parser will parse the firewall/pf logs. You can have it parse more, such as Suricata, and in this case it would be up to you to configure Suricata to only look at WAN or to have CrowdSec collect the logs and apply that filter logic in the acquis details and follow-up pre/post processing configs respectively.

The OPNSense CrowdSec plug-in also includes a Blocker Agent, it will listen to your LAPI (the Server side of your local CrowdSec plug-in) and update the WAN only blocklist the is configured as a part of the plug-in installation. This already meets your needs from what I understand.

!! Major extra / might not be on your focus !!:

You can do more to modify and retain your modification for the CrowdSec plug-in btw...

From using an external LAPI, to not using the Blocker Agent (keeping only the Parser Agent active on the OPNSense)

Then, making your own Alias and Firewall rules to use the CrowdSec list where and how you want

I have not published my how to on how to do this, as, it isn't really as good as I'd like it to be (it works but on a 10 second scale of update, and updates/refresh to the Alias active content, has took 7 seconds in the past) so once I learn how to update the data in the PF alias list on the back end of OPNSense... I'll post a blog entry on doing more with the CrowdSec feature. Likely I just need to look more into doing a manual install of CrowdSec's FreeBSD blocker on an OPNSense.

For your KEA DHCP, you are using IPv4 and IPv6?

Could you share if you are using DHCP Leases to track hostnames, or registering your hostnames in BIND or elsewhere?

I run my own BIND for my localdomain and I keep it tracking IPv4 and IPv6, but I'm a little strapped into ISC data/OPNSense API data, so without an extra box to test feature state transition I've been just waiting to see how it works for others first (KEA is still kinda new).

Thank you for any feedback, it is appreciated!!

Upgraded, all good

• Zenarmor(observer only mode)
• Suricata(modded)
• Crowdsec(modded)
• Wireguard
• Unbound
• ISC DHCP
• IPv4&IPv6 enabled for many zones

Huge thank you to all developers/maintainers/testers!! <3

In my experience, the script/cron-job that runs the OPNSense rule update via policy replaces several key files in the /usr/local/etc/suricata folder, and I believe threshold.conf is one of them

At the very least, when you do a rule build generally it is supposed to create or update that file from what I've learned with 'suricata-update' (the slightly more natural way to update rules). Will say, I feel like I only know the shallow end of the pool here, so, what actually happens and what is supposed to happen with threshold.conf is a bit unknown to me.

If you are wanting to get into more customization of Suricata and possibly use the natural stack of suricata to do things, possibly including using 'suricata-update' to make your rules files and such then please check out a blog post I made.

Using Suricata-Update on OPNSense

Follow this guide will have you turn off OPNSense's policy based rule update process and in this, you will get full control over the threshold.conf file. As already mentioned, how it gets created, and how it persists within the natural suricata/suricata-update space is still something I'm exploring.

If you didn't figure it out, this might help

You can create your own rules file, put namedfile.rules in /usr/local/etc/suricata/

You might add a rule like:

Code Select Expand

pass ip 10.10.10.2 any -> 10.10.10.3 any (msg:"Rule for Bypass Example 01"; bypass; sid:1000001; rev:1;)
If you edit the "installed_rules.yaml" file in the /usr/local/etc/suricata/ folder you can add your rules file to the list of "rule-files:"

There are other methods to do this, I believe you can using the GUI also setup bypass rules for hosts.

A) if I pass the data to my backend server unencrypted this allows Surricata running on LAN to scan the actual payload, therefore making it more effective.

Yes, otherwise Suricata won't be able to inspect the encrypted traffic and would minimize the effectiveness of inspection

B) The source of all requests now appears to be the firewall itself, as it's running the reverse proxy. Does this make Surricata less effective? Does this mean it could start blocking my reverse proxy? If yes, is there a way around it? Does it make a difference as listening on LAN is behind NAT anyway?

XFF is how you 'know who requested the traffic', you will need to make sure that the Reverse Proxy you setup in the OPNSense is adding the XFF and that you can inspect/account for that data - that said, make sure you setup Nginx/your-web-host to handle the redirects correctly - I had to learn about extra conf options for Nginx to operate correctly behind Traefik as I'd have really odd port issues otherwise

C) I don't see a way to configure nginx as a transparent proxy using the official plugin. Is this correct? I could use X-Forwarded-For, but this apparently doesn't work with IPS, am I right? Or does it?

It does, but... only if you set it up correctly - you can have the XFF value be replaced in the 'source' via Suricata settings and for you, that sounds like what you will want to do - this "SHOULD" tell the IPS to block the XFF address not itself. For me, I have an external system parsing a specific set of 'xffeve.json' events that does replace the 'source' IP with the 'XFF' IP and so that system (CrowdSec) will add the XFF/source to the Firewall and block it - I do not know if the IPS (using Suricata's built in blocker) will handle this variation correctly - I do not IPS, I just reactive-IDS (add to firewall)

D) In case of a detected intrusion will only the current connection be dropped or everything from that IP (reverse proxy potentially)

Again, using XFF correctly you can mitigate this

E) is there another way I could make WAF + TLS offloading + Surricata IPS work?

IMHO - it is a bit more useful to have Suricata in an IDS mode, customize the 'custom.yaml' for Suricata so it outputs an eve event log with the source IP replaced with the XFF IP, name it something like xffeve.json and have CrowdSec parse that log. CrowdSec will add the IP to your firewall in very short time, and you can setup CrowdSec Multi-Server (one Multi-Server instance is free) and protect quite a group of things so long as you are able to connect all the Agents (Parsers/Blockers/Appsec) back to your LAPI. Yes I am suggesting a whole new Plug-in (a native OPNSense one) to solve the issue - but as far as terminating TLS/SSL you must use an actual Reverse Proxy, and that means XFF is now in the picture, there is no 'transparent reverse proxy' here, that would only happen if you are not terminating TLS/SSL and doing pass-through (which you can do in some reverse proxies... but you have to have a good reason to do this - and that would leave the traffic encrypted, which you do not want...)

Just an update, this Git Repo got a pretty sizable update this week as I changed how the script interacts with BIND.

Instead of doing per item requests it now does an AXFR (Zone Transfer) for the Forward or Reverse Zones and then checks if those items should exist or not. Now it will add missing and remove extra from BIND quite successfully and completely.

If you can "drill/dig -x <ip_address>" (drill or dig, whichever you have installed) or "drill/dig <hostname>.<localdomain>.<tld>" and get expected values back, then yes your hosts are getting DNS set correctly and your OPNSense 'should' be able to resolve these via the Alias system

If that isn't working, you might want to make sure that your OPNSense is using the DNS you are, you can tell the OPNSense not to use its internal DNS system and use whatever you have set in System -> Settings -> General by check marking the "Do not use the local DNS service as a nameserver for this system" option on that page. In short, I'm not really sure why your host can resolve hosts on your network but the OPNSense seems to be struggling there for you - I do know that I've found IPv6 to be skipped a fair amount by OPNSense Reverse and by some accounts even Forward resolution. To fix this, I setup my own BIND and have both IPv4 and IPv6 getting updated into it via both the DHCPv4 server and a Python script that scrapes all the sources of IP data and updates my BIND as necessary.

https://github.com/j0nny55555/homelabdnsupdater

Please feel free to detail a bit more about what is replying your host resolutions correctly inside your network (is it the OPNSense, or something else?), and remember, you can dig/drill @ip_address so you can specify who you ask about a local DNS resolution...

Code Select Expand

drill myhost.homelab.home AAAA @ns02.homelab.home - Forward Lookup (specifically IPv6 - quad-A)

Code Select Expand

drill -x 192.168.0.12 @ns02.homelab.home - Reverse Lookup

if you use your own BIND, and have alias doing internal resolution, and also have Overrides setup for say Unbound...

then OPNSense will use Unbound's data for the hosts you have overridden, the rest will resolve via your BIND

this is both a curse and a gift, if you have infra that needs to come up on boot, you can have the OPNSense Unbound Overrides fill-in for those parts, then the rest will come up and be completely automated...

you can API the aliases... and I'll bet you can API to Unbound's settings but I have not explored that yet