Messages

[This applies specifically to OPNsense Suricata IPS using Divert mode + Hyperscan. Do not use this workaround for normal Netmap/IDS installations unless you understand why you need it.]

Hi,

I am running OPNsense:

OPNsense 26.7.2_2-amd64
FreeBSD 15.1-RELEASE-p2
OpenSSL 3.5.7

With Suricata IPS in Divert mode and Hyperscan.

There is currently an open OPNsense issue describing an intermittent deadlock during a live Suricata rule reload with SIGUSR2 in this configuration:
https://github.com/opnsense/core/issues/10416

The stock scheduled IDS rule update currently follows this path:

rule-updater.py installRules.py pkill -USR2 suricata

The workaround described in the issue is to update/install the rules and then perform a full Suricata restart instead of entering the problematic live-reload path.

I wanted to keep automatic rule updates, but add some safeguards around that workaround.

The wrapper below prevents concurrent runs with lockf, uses OPNsense's own rule updater and installer, backs up the active ruleset, skips an unnecessary restart when the active rules did not change, validates the new rules before restarting, performs a full restart without SIGUSR2, checks whether the Divert listeners return, and restores the previous active ruleset if validation/startup fails.

This is a temporary workaround, not an upstream fix.

1. Create the update wrapper

Create:
/usr/local/sbin/suricata-safe-update

contents:

Code Select Expand

#!/bin/sh

set -u

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
export PATH

TAG="suricata-safe-update"

UPDATER="/usr/local/opnsense/scripts/suricata/rule-updater.py"
INSTALLER="/usr/local/opnsense/scripts/suricata/installRules.py"
SURICATA="/usr/local/bin/suricata"
CONFIG="/usr/local/etc/suricata/suricata.yaml"
RC="/usr/local/etc/rc.d/suricata"

ACTIVE_DIR="/usr/local/etc/suricata/opnsense.rules"
ACTIVE_YAML="/usr/local/etc/suricata/installed_rules.yaml"

BACKUP_DIR=""

logmsg()
{
    logger -t "$TAG" "$*"
    echo "$TAG: $*"
}

active_hash()
{
    {
        if [ -d "$ACTIVE_DIR" ]; then
            find "$ACTIVE_DIR" -type f -name '*.rules' -print 2>/dev/null |
                sort |
                while IFS= read -r f; do
                    printf '%s ' "$f"
                    sha256 -q "$f"
                done
        fi

        if [ -f "$ACTIVE_YAML" ]; then
            printf '%s ' "$ACTIVE_YAML"
            sha256 -q "$ACTIVE_YAML"
        fi
    } | sha256 -q
}

divert_count()
{
    sockstat 2>/dev/null |
        awk '$2 == "suricata" && $5 == "div4" && $6 == "*:8000" { n++ }
             END { print n + 0 }'
}

restore_active()
{
    logmsg "restoring previous active ruleset"

    rm -rf "$ACTIVE_DIR"

    if [ -d "$BACKUP_DIR/opnsense.rules" ]; then
        cp -a "$BACKUP_DIR/opnsense.rules" "$ACTIVE_DIR"
    fi

    if [ -f "$BACKUP_DIR/installed_rules.yaml" ]; then
        cp -p "$BACKUP_DIR/installed_rules.yaml" "$ACTIVE_YAML"
    else
        rm -f "$ACTIVE_YAML"
    fi
}

cleanup()
{
    if [ -n "${BACKUP_DIR:-}" ] && [ -d "$BACKUP_DIR" ]; then
        rm -rf "$BACKUP_DIR"
    fi
}

trap cleanup EXIT HUP INT TERM

for f in "$UPDATER" "$INSTALLER" "$SURICATA" "$CONFIG" "$RC"; do
    if [ ! -e "$f" ]; then
        logmsg "ABORT: required file missing: $f"
        exit 10
    fi
done

if ! "$RC" status >/dev/null 2>&1; then
    logmsg "ABORT: Suricata is not running"
    exit 11
fi

OLD_DIVERT=$(divert_count)

if [ "$OLD_DIVERT" -lt 1 ]; then
    logmsg "ABORT: no active Suricata Divert listeners found"
    exit 12
fi

OLD_PID=$(cat /var/run/suricata.pid 2>/dev/null || true)
BEFORE=$(active_hash)

logmsg "starting rule update; current PID=$OLD_PID divert_listeners=$OLD_DIVERT"

"$UPDATER"
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "FAILED: rule-updater.py rc=$rc"
    exit "$rc"
fi

BACKUP_DIR=$(mktemp -d /root/suricata-safe-update.XXXXXX)

if [ -d "$ACTIVE_DIR" ]; then
    cp -a "$ACTIVE_DIR" "$BACKUP_DIR/opnsense.rules"
fi

if [ -f "$ACTIVE_YAML" ]; then
    cp -p "$ACTIVE_YAML" "$BACKUP_DIR/installed_rules.yaml"
fi

logmsg "installing updated rules"

"$INSTALLER"
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "FAILED: installRules.py rc=$rc"
    restore_active
    exit "$rc"
fi

AFTER=$(active_hash)

if [ "$BEFORE" = "$AFTER" ]; then
    logmsg "no active rule changes; restart skipped"
    exit 0
fi

logmsg "active rules changed; validating new ruleset"

"$SURICATA" -T --init-errors-fatal -c "$CONFIG"
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "FAILED: Suricata validation rc=$rc; restart NOT performed"
    restore_active
    exit "$rc"
fi

logmsg "validation passed; performing full Suricata restart"

"$RC" restart
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "FAILED: restart rc=$rc; restoring previous ruleset"
    restore_active
    "$RC" restart
    exit 30
fi

i=0
while [ "$i" -lt 36 ]; do
    sleep 5

    NEW_DIVERT=$(divert_count)

    if "$RC" status >/dev/null 2>&1 &&
       [ "$NEW_DIVERT" -eq "$OLD_DIVERT" ]; then
        NEW_PID=$(cat /var/run/suricata.pid 2>/dev/null || true)
        logmsg "SUCCESS: PID=$NEW_PID divert_listeners=$NEW_DIVERT"
        exit 0
    fi

    i=$((i + 1))
done

logmsg "FAILED: Divert listeners did not recover within 180 seconds"
logmsg "restoring previous ruleset and restarting"

restore_active
"$RC" restart
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "CRITICAL: rollback restart failed rc=$rc"
    exit 40
fi

logmsg "ROLLBACK completed; manual verification required"
exit 41
Set ownership and permissions:

Code Select Expand

chown root:wheel /usr/local/sbin/suricata-safe-update
chmod 0700 /usr/local/sbin/suricata-safe-update
Check shell syntax:

Code Select Expand

/bin/sh -n /usr/local/sbin/suricata-safe-update
Optional check that the wrapper itself does not contain a SIGUSR2, pkill, or kill command:

Code Select Expand

grep -nEi 'USR2|pkill|kill ' /usr/local/sbin/suricata-safe-update
No output is expected.

2. Create a configd action

Create:
/usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.conf

Contents:

Code Select Expand

[update]
command:/usr/bin/lockf -k -t 0 /var/run/suricata-safe-update.lock /usr/local/sbin/suricata-safe-update
parameters:
type:script
message:safely update Suricata rules with full restart when needed
description:Suricata safe rule update (full restart, no SIGUSR2)
Set permissions:

Code Select Expand

chown root:wheel /usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.conf
chmod 0644 /usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.conf
Reload configd:

Code Select Expand

service configd restart
Check that the action was registered:

Code Select Expand

configctl configd actions | grep -i suricatasafe
Expected:
suricatasafe update [ Suricata safe rule update (full restart, no SIGUSR2) ]

3. Test manually before changing Cron

Check the current Suricata PID:

Code Select Expand

cat /var/run/suricata.pid
Check the current Suricata sockets:

Code Select Expand

sockstat | grep '[s]uricata'
For the actual test I recommend using detached configctl:

Code Select Expand

/usr/local/sbin/configctl -d -- suricatasafe update
A synchronous call may exceed the configctl client timeout because Hyperscan validation and startup can take several minutes.

Monitor the updater and Suricata:

Code Select Expand

ps -axww -o pid,ppid,state,etime,%cpu,%mem,rss,vsz,command | grep -E 'suricata-safe-update|suricata -T|/usr/local/bin/suricata' | grep -v grep
Check the sockets:

Code Select Expand

sockstat | grep '[s]uricata'
After the update has finished, check the PID again:

Code Select Expand

cat /var/run/suricata.pid
Check whether the new engine completed startup:

Code Select Expand

/bin/sh -c 'PID=$(cat /var/run/suricata.pid); grep "suricata $PID" /var/log/suricata/suricata_*.log | grep -E "Engine started|<Error>|<Warning>" | tail -30'
A successful start should contain something similar to:
Threads created -> W: 8 FM: 1 FR: 1  Engine started.

The number of workers/listeners depends on the system configuration.

4. Replace the normal scheduled IDS update

Go to:

System → Settings → Cron

Disable the existing:
ids rule updates

I kept the original job present but disabled, rather than deleting it, so reverting is easy.

Then create a new Cron job using:
Suricata safe rule update (full restart, no SIGUSR2)
For example, daily at 02:00:

Minutes:  0 Hours:    2 Days:      * Months:    * Weekdays:  *

OPNsense should generate an effective cron entry similar to:

Code Select Expand

0 2 * * * /usr/local/sbin/configctl -d -- suricatasafe update
Verify the effective cron:

Code Select Expand

grep -nE 'ids update|suricatasafe update' /var/cron/tabs/nobody
The old active command:

Code Select Expand

/usr/local/sbin/configctl -d -- ids update
should be gone.

The new active command should be:

Code Select Expand

/usr/local/sbin/configctl -d -- suricatasafe update
5. Why detached configctl -d?

On my system a real rule update plus Hyperscan validation and full Suricata restart takes longer than a synchronous configctl client waits.

During testing, a synchronous invocation produced:

error in configd communication TimeoutError: timed out

The configd action itself continued running and completed successfully.

For scheduled execution I therefore use:

Code Select Expand

/usr/local/sbin/configctl -d -- suricatasafe update
The lockf around the wrapper prevents two update jobs from running concurrently.

6. Tested result

Tested on:

OPNsense:        26.7.2_2 Suricata:        8.0.6 IPS mode:        Divert Pattern matcher: Hyperscan

A real rule update resulted in a full Suricata restart.

During startup the new Suricata process spent roughly two minutes at high CPU while compiling/initializing the rules.

Afterwards all eight Divert listeners on my system returned:

div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000

The log then reported:
Threads created -> W: 8 FM: 1 FR: 1  Engine started.

SHA256 of the exact wrapper version I tested:

93d4235d5c79a3ce2aba22bbc80e049601ca0d5ca88a845f24afb67ed928231f

7. Important note

This is specifically a workaround for the current Divert + Hyperscan live-reload problem described in:

OPNsense core issue #10416

The automatic path deliberately avoids:

Code Select Expand

pkill -USR2 suricata
and performs a full restart when the active ruleset changes.

A full restart introduces a short inspection interruption while Suricata starts again, so anyone using this should evaluate that behavior for their own inline setup.

The wrapper above also checks OPNsense Divert listeners on:
*:8000

because that is the Divert configuration used on my system. Anyone with a substantially different setup should review the listener detection before copying it unchanged.

This is not intended to replace Suricata's normal live-reload mechanism permanently. Once the upstream Divert/Hyperscan issue is fixed and that fix is available in OPNsense, I intend to remove this workaround and return to the stock OPNsense rule-update path.

I hope this helps somebody else also.

This applies specifically to OPNsense Suricata IPS using Divert mode + Hyperscan. Do not use this workaround for normal Netmap/IDS installations unless you understand why you need it.

I just made the jump with the outbound NAT to SNAT conversion using the NAT migration tool.

This is working as expected! Thanks for that. One minor human error occurred here. When I imported the CSV file, there was a validation error with one of the (non-activated) old outbound NAT rules.
I ignored this warning because it is no longer in use.

I clicked the [green checkmark] and the window with this validation error stayed open. I clicked the green checkmark two more times and then decided to click the [ x ]to close it.

I noticed that my SNAT has now been populated with three times the same rules ;-). Maybe a control function can be added to the migration tool to prevent importing the same rules repeatedly if someone accidentally clicks the green checkmark more than once?

Thanks, that works.

@yourfriendarmando thanks for this. I like the color's with categories. Never used that before. I can follow your guide until the [Firewall :: Rules [New] or classic, same concept] part. (I use the [New])

Can you write the firewall rules out so it is clear what belongs to what part of the firewall-rule?

And [local_link] is missing in your guide(?)


N.B> with NAT: I assumed the Destination ::
  Invert Source:  [X] ->should be-> Invert Destination: [X]
  Source Address: This Firewall -> Destination Address: This Firewall

Yeah qfeeds was enabled but it is blocking it again. I noticed qfeeds blocks legitimatie site/servers more often. Hope it gets better from user input.

Thanks, and correct. Website is working again.

another false positive. you seem to block https://internet.nl/ their ip.
As soon when I disable Q-feeds it is working.

Hi, good ask. From my point of view there is not one way to go. There are multiple roads to follow, just what you like most.
I'm no pro on this topic, but after my extended search/reading/trying; I came to this setup:

Opnsense with Adguard Home plugin + as upstream DNS Opnsense Bind (with DNSSEC) (with NO DNS Forwarders)

This way only the DNS Root servers get queried, and not one DNS server has all your queries, most privacy other than with DoH DoT DNSCrypt.

to be sure: After you changed the hit from allow to drop, you have to go to [intrusion detection - administration - rules] and hit [Apply]
have you done that?

+1 i'm curious about the developments also. This is very useful when using an open wifi, e.g. at the airport, and not being able to use a vpn on it to securely connect to your home devices, email etc. 

I still have the CSV file that was exported and imported. In this file this rule isn't there. (And nobody would like to have such a rule ;-))

Makes no sense to me. What does this dump?

# pluginctl -g filter.rule


Cheers,
Franco

when I run this now I get:

pluginctl -g filter.rule
[]

this was in the export file I made of the old rules with the added magic appeared rule, before deleting it:

Code Select Expand

@uuid,enabled,statetype,state-policy,sequence,action,quick,interfacenot,interface,direction,ipprotocol,protocol,icmptype,icmp6type,gateway,replyto,disablereplyto,log,allowopts,nosync,nopfsync,statetimeout,max-src-nodes,max-src-states,max-src-conn,max,max-src-conn-rate,max-src-conn-rates,overload,adaptivestart,adaptiveend,prio,set-prio,set-prio-low,tag,tagged,tcpflags1,tcpflags2,categories,sched,tos,shaper1,shaper2,description,source_not,source_net,source_port,destination_not,destination_net,destination_port
3af43003-284b-4680-ab3f-faffe9391068,1,keep,,1,pass,1,0,opt3,in,inet46,any,,,,,0,0,0,0,0,,,,,,,,,,,,,,,,,,,,,,,,0,any,,0,any,
I think the export / import tool seems to do something when there is an error mentioned. If I interpreted the various notifications around this on the forum correctly.

This looks related also:
https://forum.opnsense.org/index.php?topic=50651.0

I have my imported CSV list still here and looked through them. There is no allow all rule there.
Since I have all the rules with a description it was easy to see that there was none without one like the screen capture above.
When searching for WAN I did not find an allow all rule.

Maybe you can replicate this also for this out of the blue rule.

Found it! Some little bug. Thanks Patrick.
Your simple "there must be some rule allowing this" made me wonder if the deleting of the old rules has done its job or not.

And there I went through the old interface rules and there was one rule left on WAN! So the delete all (old)rules with [Remove all legacy rules] in the wizard, did not do it all. Maybe a bug there? The wizard forgot to remove one by rather just adding an important one you do not want to have!

IPv4+6 *    *    *    *    *    *    *