Suricata rule modifications via suricata-update

[jonny5]

Goal:
To modify specific SIDs using suricata-update's "--modify" option

To do this we need suricata-update, and it is already installed w/Suricata! ^_^ Nice!!

Now, it has a lot of flags/options, and OPNSense's Suricata installation is a bit unique, so, after a bit of discovery I think I've resolve this to be the most accurate form of the command:

Code: [Select]

suricata-update --suricata-conf /usr/local/etc/suricata/suricata.yaml --suricata /usr/local/bin/suricata --data-dir /usr/local/etc/suricata --no-merge --modify-conf=/root/suricata/modify.conf --output /usr/local/etc/suricata/rules --no-test --no-reload --offline
This appears to update/replace files in the /usr/local/suricata/rules folder, but, it does not make a sid map file in the same format (v1 and v2 both look different than the one OPNSense sets up) and OPNSense already does things for IDS Rules in a two different main fashions...

The two fashions are defined in this file:

Code: [Select]

/usr/local/opnsense/service/conf/actions.d/actions_ids.conf
In it are the update and reload details and a few other actions. The update (update) and reload (install) scrips are as such:

Rule Updater by OPNSense

Code: [Select]

/usr/local/opnsense/scripts/suricata/rule-updater.py( gets updated rules from the internet, puts them in /usr/local/etc/suricata/rules )

Rule Installer by OPNSense

Code: [Select]

/usr/local/opnsense/scripts/suricata/installRules.py( gets rules from /usr/local/etc/suricata/rules, and appears to copy/mod them to /usr/local/etc/suricata/opnsense.rules and then makes /usr/local/etc/suricata/installed_rules.yaml - it seems to also make an SQLite file for the SIDs )

What I'm not sure about is, what might break if I get in the mix here, and...

• Disable OPNSense's Update & Reload (Install) Cron routines
• Setup new Cron Scripts in /usr/local/opnsense/service/conf/actions.d with 'Descriptions' so I can Cron them
• Have these scripts run the suricata-update with the correct flags, and create the expected merged file?

[jonny5]

For those that haven't used oinkmaster or pulled-pork, you can more or less replace text and other modifications based on the modify file - now for example, that "specific device" that does something dumb and hits a good rule (but is an accepted risk), you can exclude that specific IP with a modification to the "$HOME_NET" text, and make the rule say "[$HOME_NET, !192.168.0.15]" instead (for that area, keeping all other text in the rule).

This modification functionality does not appear available in OPNSense, but the Policy Management design is okay, but honestly, we can do the same thing with suricata-update and more. Food for thought, curious if anyone else is interested in this?

[someone]

why are you not just using the rules updater

[someone]

you cannot change the suricata yaml, it will be overwritten on reboot

[someone]

You can download the rules , modify them, then put them back in opnsense, they will be overwritten when you do a rule update, so keep a copy. They are overwritten as a security measure which is very good.

[jonny5]

If you were to make a file such as:

Code: [Select]

/usr/local/opnsense/service/conf/actions.d/actions_homelab.conf
You could add this code to it:

Code: [Select]

[configreload]                                                                 
command: /root/suricatamod.sh; exit 0                                           
parameters:                                                                     
type:script                                                                     
message:copy over and reload intrusion detection custom yaml                   
description:Copy over and reload intrusion detection custom yaml
The shell script would now be able to be scheduled by OPNSense's Web GUI Cron menu

Code: [Select]

/root/suricatamod.shFurther, let's say that script does a thing:

Code: [Select]

#!/bin/sh

# Get current date and time
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")

# Define file paths
ROOT_CUSTOM1="/root/custom.yaml"
SURICATA_CUSTOM1="/usr/local/etc/suricata/custom.yaml"
ROOT_CUSTOM2="/root/installed_rules.yaml"
SURICATA_CUSTOM2="/usr/local/etc/suricata/installed_rules.yaml"
echo "$TIMESTAMP: Checking for configuration updates..." > /root/suricatasame.log

script_name="rule-updater.py"
# Check if the script is running using ps
ps aux | grep "$script_name" | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
  echo "$TIMESTAMP: Script '$script_name' is already running." >> /root/suricatarestart.log
  exit 0 # Exit with a 0
else
  echo "$TIMESTAMP: Script '$script_name' is not running. Proceeding..." >> /root/suricatasame.log
fi

script_name="installRules.py"
# Check if the script is running using ps
ps aux | grep "$script_name" | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
  echo "$TIMESTAMP: Script '$script_name' is already running." >> /root/suricatarestart.log
  exit 0 # Exit with a 0
else
  echo "$TIMESTAMP: Script '$script_name' is not running. Proceeding..." >> /root/suricatasame.log
fi

RESTART_NEEDED="NO"
# Check if files are identical
if cmp -s "$ROOT_CUSTOM1" "$SURICATA_CUSTOM1"; then
  echo "$TIMESTAMP: $ROOT_CUSTOM1 Files are identical." >> suricatasame.log
else
  echo "$TIMESTAMP: Files are different, copying $ROOT_CUSTOM1 to $SURICATA_CUSTOM1" >> /root/suricatarestart.log
  cp "$ROOT_CUSTOM1" "$SURICATA_CUSTOM1"
  RESTART_NEEDED="YES"
fi
# Check if files are identical
if cmp -s "$ROOT_CUSTOM2" "$SURICATA_CUSTOM2"; then
  echo "$TIMESTAMP: $ROOT_CUSTOM2 Files are identical." >> suricatasame.log
else
  echo "$TIMESTAMP: Files are different, copying $ROOT_CUSTOM2 to $SURICATA_CUSTOM2" >> /root/suricatarestart.log
  cp "$ROOT_CUSTOM2" "$SURICATA_CUSTOM2"
  RESTART_NEEDED="YES"
fi
if [ "$RESTART_NEEDED" == "YES" ]; then
  service suricata restart
  echo "$TIMESTAMP: Suricata service restarted." >> /root/suricatarestart.log
fi 

exit 0
Now, the result is you get a Suricata installation that can use a (and keeps using) very expressive custom.yaml file, such as this one where I have new vars address-groups and the like:
https://www.nova-labs.net/homelab-opnsense-crowdsec-multi-server/

[jonny5]

In short OPNSense's rule management has got me quite far... but I might be ready for a rather larger logic/control application.

See here:
https://suricata-update.readthedocs.io/en/latest/update.html#modifying-rules

https://suricata-update.readthedocs.io/en/latest/update.html#example-configuration-to-modify-rules-modify-conf

OPNSense has suricata-update already, and if we can safely mod away/quite the OPNSense SID management, and/or, somehow in-between-mod the rule set. Huge win! There are a few hosts that trip rules that if I could filter that rule away from that host, it would be the best win.

On a few rules replacing $HOME_NET with [$HOME_NET, ![192.168.0.20,192.168.0.21]] (for example) would make thing great ^_^

[franco]

You can modify HOME_NET via advanced settings, you know?


Cheers,
Franco

[jonny5]

That helps, but, I would want to be able to edit that whole "vars" start part. To add a new group, make it a sub group of "$HOME_NET", etc. Then later in the modded rules, I could use specific sub groups that are already 'cleaned lists' of IPs that I want that rule to respond to.

The deal is, where I will want to remove one or more IPs from one rule's "$HOME_NET" there could be a list of other combinations/different filters for different rules. Each SID is different and some devices do dumb things but you want to catch the next dumb thing but filter the current found dumb thing away but still be aware of it at the perimeter (HTTP on not HTTP port for example).

That is where the modify.conf and suricata-update come in. You can get your rules updated, then mod them according to a matching pattern (use the --modify feature of suricata-update, there are more and there's an order they execute in) and have a customized fresh set of rules installed. I used to do this with Snort, but it was called oinkmaster (and there was pulled-pork others used), on IDS installations around the world at the previous job (not too many, it was a start-up - still 50+ IDS installations watching attacks and managing all the events and double-checking new/recently seen events (we captured the Packets on each rule using a mod the boss made, this allowed us to verify the SID worked as expected... just gotta read HEX/etc from the packet)).

Also, thank you so very much for OPNSense!! The flexibility, features, and integration, also the community, it is all why I am here. Thank you to all the support and testers too!!

[jonny5]

Okay, it happened, I've been able to:

• Turn off the built-in OPNSense update IDS rule Cron
• Add and enable two Web GUI Cron entries to the configd files
• Have the two shell scripts for those keep the custom.yaml in line and run suricata-update

It seems suricata-update takes a very long time, 30+ minutes at times to complete on an 8 core AMD processor.

So I posted on Suricata's forum to check it out, this is the trail of a future Blog/Post-update about how to use 'suricata-update' and/or Aristotle2 on your rules and only see the Alerts/Drops you would expect. I do not have all the instructions as it isn't done cooking yet. ^_^

https://forum.suricata.io/t/slow-suricata-update-on-an-opnsense-router-takes-30-minutes-for-200k-rules/5068

[jonny5]

Here is the how-to on getting OPNSense to get IDS rule updates via suricata-update and allowing rule modifications.

https://www.nova-labs.net/using-suricata-update-on-opnsense/

Note, currently for a very large disable/enable/drop/modify lists it could take 30+ minutes for a rule update to complete. There is an upcoming fix to resolve this mentioned here - https://forum.suricata.io/t/slow-suricata-update-on-an-opnsense-router-takes-30-minutes-for-200k-rules/5068/9?u=jonny5.

Just want to mention again that the OPNSense Policy based Rule Update is quite fast (currently faster than suricata-update in most cases) and good, the above blog post how-to is only for advanced users that are wanting to get quite detailed/customized with their IDS/Suricata configuration.

Ideally I can mature this idea/mod into a natural feature in OPNSense (especially after the updated suricata-update comes out).