Goal:To modify specific SIDs using suricata-update's "--modify" option
To do this we need suricata-update, and it is already installed w/Suricata! ^_^ Nice!!
Now, it has a lot of flags/options, and OPNSense's Suricata installation is a bit unique, so, after a bit of discovery I think I've resolve this to be the most accurate form of the command:
suricata-update --suricata-conf /usr/local/etc/suricata/suricata.yaml --suricata /usr/local/bin/suricata --data-dir /usr/local/etc/suricata --no-merge --modify-conf=/root/suricata/modify.conf --output /usr/local/etc/suricata/rules --no-test --no-reload --offlineThis appears to update/replace files in the /usr/local/suricata/rules folder, but, it does not make a sid map file in the same format (v1 and v2 both look different than the one OPNSense sets up) and OPNSense already does things for IDS Rules in a two different main fashions...
The two fashions are defined in this file:
/usr/local/opnsense/service/conf/actions.d/actions_ids.conf In it are the update and reload details and a few other actions. The update (update) and reload (install) scrips are as such:
Rule Updater by OPNSense/usr/local/opnsense/scripts/suricata/rule-updater.py( gets updated rules from the internet, puts them in /usr/local/etc/suricata/rules )
Rule Installer by OPNSense/usr/local/opnsense/scripts/suricata/installRules.py( gets rules from /usr/local/etc/suricata/rules, and appears to copy/mod them to /usr/local/etc/suricata/opnsense.rules and then makes /usr/local/etc/suricata/installed_rules.yaml - it seems to also make an SQLite file for the SIDs )
What I'm not sure about is, what might break if I get in the mix here, and...
- Disable OPNSense's Update & Reload (Install) Cron routines
- Setup new Cron Scripts in /usr/local/opnsense/service/conf/actions.d with 'Descriptions' so I can Cron them
- Have these scripts run the suricata-update with the correct flags, and create the expected merged file?
For those that haven't used oinkmaster or pulled-pork, you can more or less replace text and other modifications based on the modify file - now for example, that "specific device" that does something dumb and hits a good rule (but is an accepted risk), you can exclude that specific IP with a modification to the "$HOME_NET" text, and make the rule say "[$HOME_NET, !192.168.0.15]" instead (for that area, keeping all other text in the rule).
This modification functionality does not appear available in OPNSense, but the Policy Management design is okay, but honestly, we can do the same thing with suricata-update and more. Food for thought, curious if anyone else is interested in this?
why are you not just using the rules updater
you cannot change the suricata yaml, it will be overwritten on reboot
You can download the rules , modify them, then put them back in opnsense, they will be overwritten when you do a rule update, so keep a copy. They are overwritten as a security measure which is very good.
If you were to make a file such as:
/usr/local/opnsense/service/conf/actions.d/actions_homelab.conf
You could add this code to it:
[configreload]
command: /root/suricatamod.sh; exit 0
parameters:
type:script
message:copy over and reload intrusion detection custom yaml
description:Copy over and reload intrusion detection custom yaml
The shell script would now be able to be scheduled by OPNSense's Web GUI Cron menu
/root/suricatamod.sh
Further, let's say that script does a thing:
#!/bin/sh
# Get current date and time
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")
# Define file paths
ROOT_CUSTOM1="/root/custom.yaml"
SURICATA_CUSTOM1="/usr/local/etc/suricata/custom.yaml"
ROOT_CUSTOM2="/root/installed_rules.yaml"
SURICATA_CUSTOM2="/usr/local/etc/suricata/installed_rules.yaml"
echo "$TIMESTAMP: Checking for configuration updates..." > /root/suricatasame.log
script_name="rule-updater.py"
# Check if the script is running using ps
ps aux | grep "$script_name" | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
echo "$TIMESTAMP: Script '$script_name' is already running." >> /root/suricatarestart.log
exit 0 # Exit with a 0
else
echo "$TIMESTAMP: Script '$script_name' is not running. Proceeding..." >> /root/suricatasame.log
fi
script_name="installRules.py"
# Check if the script is running using ps
ps aux | grep "$script_name" | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
echo "$TIMESTAMP: Script '$script_name' is already running." >> /root/suricatarestart.log
exit 0 # Exit with a 0
else
echo "$TIMESTAMP: Script '$script_name' is not running. Proceeding..." >> /root/suricatasame.log
fi
RESTART_NEEDED="NO"
# Check if files are identical
if cmp -s "$ROOT_CUSTOM1" "$SURICATA_CUSTOM1"; then
echo "$TIMESTAMP: $ROOT_CUSTOM1 Files are identical." >> suricatasame.log
else
echo "$TIMESTAMP: Files are different, copying $ROOT_CUSTOM1 to $SURICATA_CUSTOM1" >> /root/suricatarestart.log
cp "$ROOT_CUSTOM1" "$SURICATA_CUSTOM1"
RESTART_NEEDED="YES"
fi
# Check if files are identical
if cmp -s "$ROOT_CUSTOM2" "$SURICATA_CUSTOM2"; then
echo "$TIMESTAMP: $ROOT_CUSTOM2 Files are identical." >> suricatasame.log
else
echo "$TIMESTAMP: Files are different, copying $ROOT_CUSTOM2 to $SURICATA_CUSTOM2" >> /root/suricatarestart.log
cp "$ROOT_CUSTOM2" "$SURICATA_CUSTOM2"
RESTART_NEEDED="YES"
fi
if [ "$RESTART_NEEDED" == "YES" ]; then
service suricata restart
echo "$TIMESTAMP: Suricata service restarted." >> /root/suricatarestart.log
fi
exit 0
Now, the result is you get a Suricata installation that can use a (and keeps using) very expressive custom.yaml file, such as this one where I have new vars address-groups and the like:
https://www.nova-labs.net/homelab-opnsense-crowdsec-multi-server/
In short OPNSense's rule management has got me quite far... but I might be ready for a rather larger logic/control application.
See here:
https://suricata-update.readthedocs.io/en/latest/update.html#modifying-rules
https://suricata-update.readthedocs.io/en/latest/update.html#example-configuration-to-modify-rules-modify-conf
OPNSense has suricata-update already, and if we can safely mod away/quite the OPNSense SID management, and/or, somehow in-between-mod the rule set. Huge win! There are a few hosts that trip rules that if I could filter that rule away from that host, it would be the best win.
On a few rules replacing $HOME_NET with [$HOME_NET, ![192.168.0.20,192.168.0.21]] (for example) would make thing great ^_^
You can modify HOME_NET via advanced settings, you know?
Cheers,
Franco
That helps, but, I would want to be able to edit that whole "vars" start part. To add a new group, make it a sub group of "$HOME_NET", etc. Then later in the modded rules, I could use specific sub groups that are already 'cleaned lists' of IPs that I want that rule to respond to.
The deal is, where I will want to remove one or more IPs from one rule's "$HOME_NET" there could be a list of other combinations/different filters for different rules. Each SID is different and some devices do dumb things but you want to catch the next dumb thing but filter the current found dumb thing away but still be aware of it at the perimeter (HTTP on not HTTP port for example).
That is where the modify.conf and suricata-update come in. You can get your rules updated, then mod them according to a matching pattern (use the --modify feature of suricata-update, there are more and there's an order they execute in) and have a customized fresh set of rules installed. I used to do this with Snort, but it was called oinkmaster (and there was pulled-pork others used), on IDS installations around the world at the previous job (not too many, it was a start-up - still 50+ IDS installations watching attacks and managing all the events and double-checking new/recently seen events (we captured the Packets on each rule using a mod the boss made, this allowed us to verify the SID worked as expected... just gotta read HEX/etc from the packet)).
Also, thank you so very much for OPNSense!! The flexibility, features, and integration, also the community, it is all why I am here. Thank you to all the support and testers too!!
Okay, it happened, I've been able to:
- Turn off the built-in OPNSense update IDS rule Cron
- Add and enable two Web GUI Cron entries to the configd files
- Have the two shell scripts for those keep the custom.yaml in line and run suricata-update
It seems suricata-update takes a very long time, 30+ minutes at times to complete on an 8 core AMD processor.
So I posted on Suricata's forum to check it out, this is the trail of a future Blog/Post-update about how to use 'suricata-update' and/or Aristotle2 on your rules and only see the Alerts/Drops you would expect. I do not have all the instructions as it isn't done cooking yet. ^_^
https://forum.suricata.io/t/slow-suricata-update-on-an-opnsense-router-takes-30-minutes-for-200k-rules/5068
Here is the how-to on getting OPNSense to get IDS rule updates via suricata-update and allowing rule modifications.
https://www.nova-labs.net/using-suricata-update-on-opnsense/ (https://www.nova-labs.net/using-suricata-update-on-opnsense/)
Note, currently for a very large disable/enable/drop/modify lists it could take 30+ minutes for a rule update to complete. There is an upcoming fix to resolve this mentioned here - https://forum.suricata.io/t/slow-suricata-update-on-an-opnsense-router-takes-30-minutes-for-200k-rules/5068/9?u=jonny5 (https://forum.suricata.io/t/slow-suricata-update-on-an-opnsense-router-takes-30-minutes-for-200k-rules/5068/9?u=jonny5).
Just want to mention again that the OPNSense Policy based Rule Update is quite fast (currently faster than suricata-update in most cases) and good, the above blog post how-to is only for advanced users that are wanting to get quite detailed/customized with their IDS/Suricata configuration.
Ideally I can mature this idea/mod into a natural feature in OPNSense (especially after the updated suricata-update comes out).
Sounds like your trying to put the suricata rules into opnsense.
Last I downloaded them and yes it takes a while, it was around 50,000 rules.
I do it manually.
I just downloaded the rules and it was around 48,000 rules
They say ET Pro is around 70000 by itself
My process now takes about 4 minutes and uses 'suricata-update' for the whole detail...
The start:
14/1/2025 -- 21:10:01 - <Debug> -- This is suricata-update version 1.3.3 (rev: None); Python: 3.11.10 (main, Nov 4 2024, 23:32:15) [Clang 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15
14/1/2025 -- 21:10:01 - <Info> -- Loading /root/suricata/update.yaml
The final stats:
14/1/2025 -- 21:13:56 - <Info> -- Disabled 5759 rules.
14/1/2025 -- 21:13:56 - <Info> -- Enabled 1821 rules.
14/1/2025 -- 21:13:56 - <Info> -- Modified 193131 rules.
14/1/2025 -- 21:13:56 - <Info> -- Dropped 0 rules.
14/1/2025 -- 21:13:56 - <Debug> -- Checking flowbits for pass 1 of rules.
14/1/2025 -- 21:13:57 - <Debug> -- Found 522 required flowbits.
14/1/2025 -- 21:13:58 - <Debug> -- Found 153 rules to enable for flowbit requirements (pass 1)
14/1/2025 -- 21:13:58 - <Debug> -- Checking flowbits for pass 2 of rules.
14/1/2025 -- 21:13:58 - <Debug> -- Found 524 required flowbits.
14/1/2025 -- 21:13:59 - <Debug> -- Found 17 rules to enable for flowbit requirements (pass 2)
14/1/2025 -- 21:13:59 - <Debug> -- Checking flowbits for pass 3 of rules.
14/1/2025 -- 21:13:59 - <Debug> -- Found 524 required flowbits.
14/1/2025 -- 21:14:00 - <Debug> -- Found 0 rules to enable for flowbit requirements (pass 3)
14/1/2025 -- 21:14:00 - <Debug> -- All required rules enabled.
14/1/2025 -- 21:14:00 - <Info> -- Enabled 169 rules for flowbit dependencies.
14/1/2025 -- 21:14:00 - <Info> -- Backing up current rules.
14/1/2025 -- 21:14:01 - <Debug> -- Recording existing file /usr/local/etc/suricata/opnsense.rules/suricata.rules with hash '3f28ee1d08f4df0fc4ba26ecca58b10d'.
14/1/2025 -- 21:14:29 - <Info> -- Writing rules to /usr/local/etc/suricata/opnsense.rules/suricata.rules: total: 193131; enabled: 138821; added: 39; removed 0; modified: 89
Finally have a good start at disable/enable/drop/modify yaml files to use with suricata-update to minimize FPs and other noisy SIDs.
Please feel free to clone/branch and suggest updates!
https://github.com/j0nny55555/noiseless-suricata-update
Quote from: franco on November 21, 2024, 09:26:45 AMYou can modify HOME_NET via advanced settings, you know?
Cheers,
Franco
Must admit, I had not seen the additional settings for IDS until today. My familiarity with the middle-ware of the OPNSense, and even some of the Web GUI specific front-ware is limited to non-existent. It would be quite a detail but possibly really great for all parties, if I could attempt to integrate my mod into the IDS management space of OPNSense. It would be best to keep both management options available, the Policy and Suricata-Update - so long as we could make that a navigable and state correct feature (ability to choose the desired management option at will) it would keep existing and add new.
Any tips, or cheat sheet or guides to designing what exists in the WebGUI + coding the dependent features in the middle-ware/backend-ware?
I would gladly take the opportunity to more or less abstract as much of the suricata.yaml + custom.yaml into options in the GUI as possible, and ideally allow for modification of the template folder's custom.yaml because as it is, what I use for my custom.yaml in raw, cannot be consumed from the template folder's custom.yaml.
The suricata rules are updated in the opnsense repository. It doesnt use or need suricata-update as far as I can tell.
All we have to do is click update rules in opnsense.
As far as rules, yes suricata has a few rules not in the opnsense rulesets, but I have not had a single hit on them yet.
The yaml is locked for security and duplication at reload
I think you can set up queues to run scripts if that is needed.
Hope that helps
Is there a problem with the rulesets?