Messages

1

Intrusion Detection and Prevention / Re: Suricata rule modifications via suricata-update

« on: December 02, 2024, 08:05:18 pm »

Here is the how-to on getting OPNSense to get IDS rule updates via suricata-update and allowing rule modifications.

https://www.nova-labs.net/using-suricata-update-on-opnsense/

Note, currently for a very large disable/enable/drop/modify lists it could take 30+ minutes for a rule update to complete. There is an upcoming fix to resolve this mentioned here - https://forum.suricata.io/t/slow-suricata-update-on-an-opnsense-router-takes-30-minutes-for-200k-rules/5068/9?u=jonny5.

Just want to mention again that the OPNSense Policy based Rule Update is quite fast (currently faster than suricata-update in most cases) and good, the above blog post how-to is only for advanced users that are wanting to get quite detailed/customized with their IDS/Suricata configuration.

Ideally I can mature this idea/mod into a natural feature in OPNSense (especially after the updated suricata-update comes out).

2

Intrusion Detection and Prevention / Re: Suricata rule modifications via suricata-update

« on: November 26, 2024, 09:50:32 pm »

Okay, it happened, I've been able to:

• Turn off the built-in OPNSense update IDS rule Cron
• Add and enable two Web GUI Cron entries to the configd files
• Have the two shell scripts for those keep the custom.yaml in line and run suricata-update

It seems suricata-update takes a very long time, 30+ minutes at times to complete on an 8 core AMD processor.

So I posted on Suricata's forum to check it out, this is the trail of a future Blog/Post-update about how to use 'suricata-update' and/or Aristotle2 on your rules and only see the Alerts/Drops you would expect. I do not have all the instructions as it isn't done cooking yet. ^_^

https://forum.suricata.io/t/slow-suricata-update-on-an-opnsense-router-takes-30-minutes-for-200k-rules/5068

3

Intrusion Detection and Prevention / Re: Proofpoint Telemetry Flowbit Issues.

« on: November 25, 2024, 05:00:09 pm »

There are only a few flowbit mentions in my logs, for anyone else tracking these are what I see with almost all rules (998 disabled of 215144 total) enabled:

To any wanting to share/check:

Code: [Select]

grep -vE '(alert|anomaly)' suricata_20241125.log | cut -w -f 10- | sort | uniq | grep flowbit
My output:

Code: [Select]

<Warning> -- flowbit 'file.doc&file.ole' is checked but not set. Checked in 17301 and 3 other sigs
<Warning> -- flowbit 'file.pdf&file.ttf' is checked but not set. Checked in 28585 and 1 other sigs
<Warning> -- flowbit 'file.ppsx&file.zip' is checked but not set. Checked in 26068 and 1 other sigs
<Warning> -- flowbit 'file.quicktime&file.swf' is checked but not set. Checked in 24672 and 0 other sigs
<Warning> -- flowbit 'file.rjs&file.zip' is checked but not set. Checked in 17461 and 0 other sigs
<Warning> -- flowbit 'file.visio&file.ole' is checked but not set. Checked in 11836 and 1 other sigs
<Warning> -- flowbit 'file.xls&file.ole' is checked but not set. Checked in 19943 and 10 other sigs
<Warning> -- flowbit 'file.xps&file.zip' is checked but not set. Checked in 45776 and 1 other sigs
<Warning> -- flowbit 'file.zip&file.silverlight' is checked but not set. Checked in 25035 and 7 other sigs
<Warning> -- flowbit 'glassfish_unauth_attempt' is checked but not set. Checked in 20160 and 0 other sigs

Pretty sure there used to be more, so I can mention that this feels like an improvement, thank you!

4

Intrusion Detection and Prevention / Re: 24 7.6: ips error in configd.py

« on: November 22, 2024, 07:27:34 am »

Also have noticed/seen the same error

It seems the rule build/move time period has expanded but I also think the log line "timeout" hits before 120 seconds has passed

5

Intrusion Detection and Prevention / Re: How to set IP for rules working

« on: November 22, 2024, 07:20:48 am »

I'm also a bit beside myself reading your messages...

The firewall appears to be working as expected

Do you have CrowdSec setup?

You don't have to manage your own block lists. They have three block lists you can subscribe to, and you can alias 'subscribe' block lists in OPNSense too.

Block lists help, but they are not a "fix". They can do as much damage as good.

Best wishes, use IDS and Firewall and enable the Firewall's list with your IDS/Suricata EVELOG Output Severity 1/2 TCP hits via CrowdSec, for free for one Security Engine

6

Intrusion Detection and Prevention / Re: Suricata rule modifications via suricata-update

« on: November 21, 2024, 04:52:18 pm »

That helps, but, I would want to be able to edit that whole "vars" start part. To add a new group, make it a sub group of "$HOME_NET", etc. Then later in the modded rules, I could use specific sub groups that are already 'cleaned lists' of IPs that I want that rule to respond to.

The deal is, where I will want to remove one or more IPs from one rule's "$HOME_NET" there could be a list of other combinations/different filters for different rules. Each SID is different and some devices do dumb things but you want to catch the next dumb thing but filter the current found dumb thing away but still be aware of it at the perimeter (HTTP on not HTTP port for example).

That is where the modify.conf and suricata-update come in. You can get your rules updated, then mod them according to a matching pattern (use the --modify feature of suricata-update, there are more and there's an order they execute in) and have a customized fresh set of rules installed. I used to do this with Snort, but it was called oinkmaster (and there was pulled-pork others used), on IDS installations around the world at the previous job (not too many, it was a start-up - still 50+ IDS installations watching attacks and managing all the events and double-checking new/recently seen events (we captured the Packets on each rule using a mod the boss made, this allowed us to verify the SID worked as expected... just gotta read HEX/etc from the packet)).

Also, thank you so very much for OPNSense!! The flexibility, features, and integration, also the community, it is all why I am here. Thank you to all the support and testers too!!

7

Intrusion Detection and Prevention / Re: Suricata rule modifications via suricata-update

« on: November 21, 2024, 08:14:13 am »

In short OPNSense's rule management has got me quite far... but I might be ready for a rather larger logic/control application.

See here:
https://suricata-update.readthedocs.io/en/latest/update.html#modifying-rules

https://suricata-update.readthedocs.io/en/latest/update.html#example-configuration-to-modify-rules-modify-conf

OPNSense has suricata-update already, and if we can safely mod away/quite the OPNSense SID management, and/or, somehow in-between-mod the rule set. Huge win! There are a few hosts that trip rules that if I could filter that rule away from that host, it would be the best win.

On a few rules replacing $HOME_NET with [$HOME_NET, ![192.168.0.20,192.168.0.21]] (for example) would make thing great ^_^

8

Intrusion Detection and Prevention / Re: Suricata rule modifications via suricata-update

« on: November 21, 2024, 08:07:22 am »

If you were to make a file such as:

Code: [Select]

/usr/local/opnsense/service/conf/actions.d/actions_homelab.conf
You could add this code to it:

Code: [Select]

[configreload]                                                                 
command: /root/suricatamod.sh; exit 0                                           
parameters:                                                                     
type:script                                                                     
message:copy over and reload intrusion detection custom yaml                   
description:Copy over and reload intrusion detection custom yaml
The shell script would now be able to be scheduled by OPNSense's Web GUI Cron menu

Code: [Select]

/root/suricatamod.shFurther, let's say that script does a thing:

Code: [Select]

#!/bin/sh

# Get current date and time
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")

# Define file paths
ROOT_CUSTOM1="/root/custom.yaml"
SURICATA_CUSTOM1="/usr/local/etc/suricata/custom.yaml"
ROOT_CUSTOM2="/root/installed_rules.yaml"
SURICATA_CUSTOM2="/usr/local/etc/suricata/installed_rules.yaml"
echo "$TIMESTAMP: Checking for configuration updates..." > /root/suricatasame.log

script_name="rule-updater.py"
# Check if the script is running using ps
ps aux | grep "$script_name" | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
  echo "$TIMESTAMP: Script '$script_name' is already running." >> /root/suricatarestart.log
  exit 0 # Exit with a 0
else
  echo "$TIMESTAMP: Script '$script_name' is not running. Proceeding..." >> /root/suricatasame.log
fi

script_name="installRules.py"
# Check if the script is running using ps
ps aux | grep "$script_name" | grep -v grep > /dev/null
if [ $? -eq 0 ]; then
  echo "$TIMESTAMP: Script '$script_name' is already running." >> /root/suricatarestart.log
  exit 0 # Exit with a 0
else
  echo "$TIMESTAMP: Script '$script_name' is not running. Proceeding..." >> /root/suricatasame.log
fi

RESTART_NEEDED="NO"
# Check if files are identical
if cmp -s "$ROOT_CUSTOM1" "$SURICATA_CUSTOM1"; then
  echo "$TIMESTAMP: $ROOT_CUSTOM1 Files are identical." >> suricatasame.log
else
  echo "$TIMESTAMP: Files are different, copying $ROOT_CUSTOM1 to $SURICATA_CUSTOM1" >> /root/suricatarestart.log
  cp "$ROOT_CUSTOM1" "$SURICATA_CUSTOM1"
  RESTART_NEEDED="YES"
fi
# Check if files are identical
if cmp -s "$ROOT_CUSTOM2" "$SURICATA_CUSTOM2"; then
  echo "$TIMESTAMP: $ROOT_CUSTOM2 Files are identical." >> suricatasame.log
else
  echo "$TIMESTAMP: Files are different, copying $ROOT_CUSTOM2 to $SURICATA_CUSTOM2" >> /root/suricatarestart.log
  cp "$ROOT_CUSTOM2" "$SURICATA_CUSTOM2"
  RESTART_NEEDED="YES"
fi
if [ "$RESTART_NEEDED" == "YES" ]; then
  service suricata restart
  echo "$TIMESTAMP: Suricata service restarted." >> /root/suricatarestart.log
fi 

exit 0
Now, the result is you get a Suricata installation that can use a (and keeps using) very expressive custom.yaml file, such as this one where I have new vars address-groups and the like:
https://www.nova-labs.net/homelab-opnsense-crowdsec-multi-server/

9

Intrusion Detection and Prevention / Re: Suricata rule modifications via suricata-update

« on: November 20, 2024, 09:12:53 am »

For those that haven't used oinkmaster or pulled-pork, you can more or less replace text and other modifications based on the modify file - now for example, that "specific device" that does something dumb and hits a good rule (but is an accepted risk), you can exclude that specific IP with a modification to the "$HOME_NET" text, and make the rule say "[$HOME_NET, !192.168.0.15]" instead (for that area, keeping all other text in the rule).

This modification functionality does not appear available in OPNSense, but the Policy Management design is okay, but honestly, we can do the same thing with suricata-update and more. Food for thought, curious if anyone else is interested in this?

10

Intrusion Detection and Prevention / Suricata rule modifications via suricata-update

« on: November 20, 2024, 09:03:42 am »

Goal:
To modify specific SIDs using suricata-update's "--modify" option

To do this we need suricata-update, and it is already installed w/Suricata! ^_^ Nice!!

Now, it has a lot of flags/options, and OPNSense's Suricata installation is a bit unique, so, after a bit of discovery I think I've resolve this to be the most accurate form of the command:

Code: [Select]

suricata-update --suricata-conf /usr/local/etc/suricata/suricata.yaml --suricata /usr/local/bin/suricata --data-dir /usr/local/etc/suricata --no-merge --modify-conf=/root/suricata/modify.conf --output /usr/local/etc/suricata/rules --no-test --no-reload --offline
This appears to update/replace files in the /usr/local/suricata/rules folder, but, it does not make a sid map file in the same format (v1 and v2 both look different than the one OPNSense sets up) and OPNSense already does things for IDS Rules in a two different main fashions...

The two fashions are defined in this file:

Code: [Select]

/usr/local/opnsense/service/conf/actions.d/actions_ids.conf
In it are the update and reload details and a few other actions. The update (update) and reload (install) scrips are as such:

Rule Updater by OPNSense

Code: [Select]

/usr/local/opnsense/scripts/suricata/rule-updater.py( gets updated rules from the internet, puts them in /usr/local/etc/suricata/rules )

Rule Installer by OPNSense

Code: [Select]

/usr/local/opnsense/scripts/suricata/installRules.py( gets rules from /usr/local/etc/suricata/rules, and appears to copy/mod them to /usr/local/etc/suricata/opnsense.rules and then makes /usr/local/etc/suricata/installed_rules.yaml - it seems to also make an SQLite file for the SIDs )

What I'm not sure about is, what might break if I get in the mix here, and...

• Disable OPNSense's Update & Reload (Install) Cron routines
• Setup new Cron Scripts in /usr/local/opnsense/service/conf/actions.d with 'Descriptions' so I can Cron them
• Have these scripts run the suricata-update with the correct flags, and create the expected merged file?

11

General Discussion / Cannot match API getRule state to Firewall Rule in GUI

« on: August 13, 2024, 11:40:03 pm »

In my Syslog output, it seems filterlog/firewall applied a simple numeric to the 'rule' and 'subrule' elements

In the API output, each rule has a UUID and I get them at

Code: [Select]

/diagnostics/firewall/listRuleIds then query the rule's elements at

Code: [Select]

/firewall/filter/getRule?{UUID} and for the output below I've summarized the output as it doesn't seem like the rule data matches what is in the GUI for OPNSense? These are some of the default rules, but, their data all says Pass and that does not make sense to me as the first four are block rules.

Code: [Select]

3c2cd03c70091e3732710e44c3b97506 named Block bogon IPv4 networks from WAN is...
['action pass Pass',
 'direction in In',
 'ipprotocol inet IPv4',
 'protocol any any',
 'gateway  None']
57401c13616c94401fc89cafa777581e named Block bogon IPv6 networks from WAN is...
['action pass Pass',
 'direction in In',
 'ipprotocol inet IPv4',
 'protocol any any',
 'gateway  None']
1072878c6245b52440bc89c6107a9d0a named Block private networks from WAN is...
['action pass Pass',
 'direction in In',
 'ipprotocol inet IPv4',
 'protocol any any',
 'gateway  None']
dcde0621a9f0daa594b014e15f65c076 named Block private networks from WAN is...
['action pass Pass',
 'direction in In',
 'ipprotocol inet IPv4',
 'protocol any any',
 'gateway  None']
5ddcbf1f0688962629f1a2166ba2ab0c named CARP defaults is...
['action pass Pass',
 'direction in In',
 'ipprotocol inet IPv4',
 'protocol any any',
 'gateway  None']
846c09139ef5484c01967052b15e454a named CARP defaults is...
['action pass Pass',
 'direction in In',
 'ipprotocol inet IPv4',
 'protocol any any',
 'gateway  None']
In the end, I am after a method to match the rule+subrule to a UUID, so that I can label my rules in my Syslog output/Kibana so that it shows the firewall histogram w/Rule naming context?

12

Intrusion Detection and Prevention / Re: How do I change suricata.yaml and get it to stick

« on: July 27, 2024, 07:44:13 am »

Generation error message example

13

Intrusion Detection and Prevention / Re: How do I change suricata.yaml and get it to stick

« on: July 27, 2024, 07:39:50 am »

While I do not have a solution, I did want to mention you can edit

Code: [Select]

/usr/local/etc/suricata/custom.yaml and then simply restart the service and have that change be used and stay for a while. You can even replace the "host-os-policy:" area here it seems, and enable additional features in "app-layer" that are normally disabled by default.

If you use the OPNSense IDS Administration GUI, set a Policy, or enable or disable a feature or Rule, the back-end actions will over write your custom.yaml file with the one found at

Code: [Select]

/usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml and luckily you can modify that file a little and have it work or at least in the past you could - I am currently having some difficulty there.

If you change the /usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml file at all it appears it will have a generation failure that shows up in the OPNSense IDS Admin GUI. If you delete the /usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml file, it will have a generation failure.

In short, currently, there is no way to do it.

Default settings are good, I want to customize some XFF output and have that stay around

A supported "custom.yaml" file where ideally you could over-write/replace all and add to suricata.yaml set options (in short you could replace most if not all the existing settings and/or add to them would be amazing.

14

24.7 Production Series / Re: 24.7 upgrade was smooth

« on: July 25, 2024, 11:35:41 pm »

Can confirm!

IPv4 and IPv6 working well, Unbound/CrowdSec/Wireguard working as expected

Have not adopted the new DHCP method(s) yet, still on the old service, to dos, to dos..

Thank you OPNSense!!

15

24.1 Legacy Series / Re: Upgrade to 24.1.2 appears to be downloading a massive base-24.1.2.txz

« on: February 20, 2024, 04:53:50 pm »

It is a rather tiny file of about 100+ Mbs, not sure what is going on...

https://pkg.opnsense.org/FreeBSD:13:amd64/24.1/sets/

Edit:
I wait for about 5 rows of "......." and I manually restarted, asked it to check for updates, and it updated the two base/kernel packages, and everything appears good to go.

Possibly unique hiccup? Not sure, but recoverable it appears!