OPNsense Forum

Hi,

I am running OPNsense:

OPNsense 26.7.2_2-amd64
FreeBSD 15.1-RELEASE-p2
OpenSSL 3.5.7

With Suricata IPS in Divert mode and Hyperscan.

There is currently an open OPNsense issue describing an intermittent deadlock during a live Suricata rule reload with SIGUSR2 in this configuration:
https://github.com/opnsense/core/issues/10416 (https://github.com/opnsense/core/issues/10416)

The stock scheduled IDS rule update currently follows this path:

rule-updater.py installRules.py pkill -USR2 suricata

The workaround described in the issue is to update/install the rules and then perform a full Suricata restart instead of entering the problematic live-reload path.

I wanted to keep automatic rule updates, but add some safeguards around that workaround.

The wrapper below prevents concurrent runs with lockf, uses OPNsense's own rule updater and installer, backs up the active ruleset, skips an unnecessary restart when the active rules did not change, validates the new rules before restarting, performs a full restart without SIGUSR2, checks whether the Divert listeners return, and restores the previous active ruleset if validation/startup fails.

This is a temporary workaround, not an upstream fix.

1. Create the update wrapper

Create:
/usr/local/sbin/suricata-safe-update

contents:
#!/bin/sh

set -u

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
export PATH

TAG="suricata-safe-update"

UPDATER="/usr/local/opnsense/scripts/suricata/rule-updater.py"
INSTALLER="/usr/local/opnsense/scripts/suricata/installRules.py"
SURICATA="/usr/local/bin/suricata"
CONFIG="/usr/local/etc/suricata/suricata.yaml"
RC="/usr/local/etc/rc.d/suricata"

ACTIVE_DIR="/usr/local/etc/suricata/opnsense.rules"
ACTIVE_YAML="/usr/local/etc/suricata/installed_rules.yaml"

BACKUP_DIR=""

logmsg()
{
    logger -t "$TAG" "$*"
    echo "$TAG: $*"
}

active_hash()
{
    {
        if [ -d "$ACTIVE_DIR" ]; then
            find "$ACTIVE_DIR" -type f -name '*.rules' -print 2>/dev/null |
                sort |
                while IFS= read -r f; do
                    printf '%s ' "$f"
                    sha256 -q "$f"
                done
        fi

        if [ -f "$ACTIVE_YAML" ]; then
            printf '%s ' "$ACTIVE_YAML"
            sha256 -q "$ACTIVE_YAML"
        fi
    } | sha256 -q
}

divert_count()
{
    sockstat 2>/dev/null |
        awk '$2 == "suricata" && $5 == "div4" && $6 == "*:8000" { n++ }
             END { print n + 0 }'
}

restore_active()
{
    logmsg "restoring previous active ruleset"

    rm -rf "$ACTIVE_DIR"

    if [ -d "$BACKUP_DIR/opnsense.rules" ]; then
        cp -a "$BACKUP_DIR/opnsense.rules" "$ACTIVE_DIR"
    fi

    if [ -f "$BACKUP_DIR/installed_rules.yaml" ]; then
        cp -p "$BACKUP_DIR/installed_rules.yaml" "$ACTIVE_YAML"
    else
        rm -f "$ACTIVE_YAML"
    fi
}

cleanup()
{
    if [ -n "${BACKUP_DIR:-}" ] && [ -d "$BACKUP_DIR" ]; then
        rm -rf "$BACKUP_DIR"
    fi
}

trap cleanup EXIT HUP INT TERM

for f in "$UPDATER" "$INSTALLER" "$SURICATA" "$CONFIG" "$RC"; do
    if [ ! -e "$f" ]; then
        logmsg "ABORT: required file missing: $f"
        exit 10
    fi
done

if ! "$RC" status >/dev/null 2>&1; then
    logmsg "ABORT: Suricata is not running"
    exit 11
fi

OLD_DIVERT=$(divert_count)

if [ "$OLD_DIVERT" -lt 1 ]; then
    logmsg "ABORT: no active Suricata Divert listeners found"
    exit 12
fi

OLD_PID=$(cat /var/run/suricata.pid 2>/dev/null || true)
BEFORE=$(active_hash)

logmsg "starting rule update; current PID=$OLD_PID divert_listeners=$OLD_DIVERT"

"$UPDATER"
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "FAILED: rule-updater.py rc=$rc"
    exit "$rc"
fi

BACKUP_DIR=$(mktemp -d /root/suricata-safe-update.XXXXXX)

if [ -d "$ACTIVE_DIR" ]; then
    cp -a "$ACTIVE_DIR" "$BACKUP_DIR/opnsense.rules"
fi

if [ -f "$ACTIVE_YAML" ]; then
    cp -p "$ACTIVE_YAML" "$BACKUP_DIR/installed_rules.yaml"
fi

logmsg "installing updated rules"

"$INSTALLER"
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "FAILED: installRules.py rc=$rc"
    restore_active
    exit "$rc"
fi

AFTER=$(active_hash)

if [ "$BEFORE" = "$AFTER" ]; then
    logmsg "no active rule changes; restart skipped"
    exit 0
fi

logmsg "active rules changed; validating new ruleset"

"$SURICATA" -T --init-errors-fatal -c "$CONFIG"
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "FAILED: Suricata validation rc=$rc; restart NOT performed"
    restore_active
    exit "$rc"
fi

logmsg "validation passed; performing full Suricata restart"

"$RC" restart
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "FAILED: restart rc=$rc; restoring previous ruleset"
    restore_active
    "$RC" restart
    exit 30
fi

i=0
while [ "$i" -lt 36 ]; do
    sleep 5

    NEW_DIVERT=$(divert_count)

    if "$RC" status >/dev/null 2>&1 &&
       [ "$NEW_DIVERT" -eq "$OLD_DIVERT" ]; then
        NEW_PID=$(cat /var/run/suricata.pid 2>/dev/null || true)
        logmsg "SUCCESS: PID=$NEW_PID divert_listeners=$NEW_DIVERT"
        exit 0
    fi

    i=$((i + 1))
done

logmsg "FAILED: Divert listeners did not recover within 180 seconds"
logmsg "restoring previous ruleset and restarting"

restore_active
"$RC" restart
rc=$?

if [ "$rc" -ne 0 ]; then
    logmsg "CRITICAL: rollback restart failed rc=$rc"
    exit 40
fi

logmsg "ROLLBACK completed; manual verification required"
exit 41
Set ownership and permissions:
chown root:wheel /usr/local/sbin/suricata-safe-update
chmod 0700 /usr/local/sbin/suricata-safe-update
Check shell syntax:
/bin/sh -n /usr/local/sbin/suricata-safe-update
Optional check that the wrapper itself does not contain a SIGUSR2, pkill, or kill command:
grep -nEi 'USR2|pkill|kill ' /usr/local/sbin/suricata-safe-update
No output is expected.

2. Create a configd action

Create:
/usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.conf

Contents:
[update]
command:/usr/bin/lockf -k -t 0 /var/run/suricata-safe-update.lock /usr/local/sbin/suricata-safe-update
parameters:
type:script
message:safely update Suricata rules with full restart when needed
description:Suricata safe rule update (full restart, no SIGUSR2)
Set permissions:
chown root:wheel /usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.conf
chmod 0644 /usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.conf
Reload configd:
service configd restart
Check that the action was registered:
configctl configd actions | grep -i suricatasafe
Expected:
suricatasafe update [ Suricata safe rule update (full restart, no SIGUSR2) ]

3. Test manually before changing Cron

Check the current Suricata PID:
cat /var/run/suricata.pid
Check the current Suricata sockets:
sockstat | grep '[s]uricata'
For the actual test I recommend using detached configctl:
/usr/local/sbin/configctl -d -- suricatasafe update
A synchronous call may exceed the configctl client timeout because Hyperscan validation and startup can take several minutes.

Monitor the updater and Suricata:
ps -axww -o pid,ppid,state,etime,%cpu,%mem,rss,vsz,command | grep -E 'suricata-safe-update|suricata -T|/usr/local/bin/suricata' | grep -v grep
Check the sockets:
sockstat | grep '[s]uricata'
After the update has finished, check the PID again:
cat /var/run/suricata.pid
Check whether the new engine completed startup:
/bin/sh -c 'PID=$(cat /var/run/suricata.pid); grep "suricata $PID" /var/log/suricata/suricata_*.log | grep -E "Engine started|<Error>|<Warning>" | tail -30'
A successful start should contain something similar to:
Threads created -> W: 8 FM: 1 FR: 1  Engine started.

The number of workers/listeners depends on the system configuration.

4. Replace the normal scheduled IDS update

Go to:

System → Settings → Cron

Disable the existing:
ids rule updates

I kept the original job present but disabled, rather than deleting it, so reverting is easy.

Then create a new Cron job using:
Suricata safe rule update (full restart, no SIGUSR2)
For example, daily at 02:00:

Minutes:  0 Hours:    2 Days:      * Months:    * Weekdays:  *

OPNsense should generate an effective cron entry similar to:
0 2 * * * /usr/local/sbin/configctl -d -- suricatasafe update
Verify the effective cron:
grep -nE 'ids update|suricatasafe update' /var/cron/tabs/nobody
The old active command:
/usr/local/sbin/configctl -d -- ids update
should be gone.

The new active command should be:
/usr/local/sbin/configctl -d -- suricatasafe update
5. Why detached configctl -d?

On my system a real rule update plus Hyperscan validation and full Suricata restart takes longer than a synchronous configctl client waits.

During testing, a synchronous invocation produced:

error in configd communication TimeoutError: timed out

The configd action itself continued running and completed successfully.

For scheduled execution I therefore use:
/usr/local/sbin/configctl -d -- suricatasafe update
The lockf around the wrapper prevents two update jobs from running concurrently.

6. Tested result

Tested on:

OPNsense:        26.7.2_2 Suricata:        8.0.6 IPS mode:        Divert Pattern matcher: Hyperscan

A real rule update resulted in a full Suricata restart.

During startup the new Suricata process spent roughly two minutes at high CPU while compiling/initializing the rules.

Afterwards all eight Divert listeners on my system returned:

div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000

The log then reported:
Threads created -> W: 8 FM: 1 FR: 1  Engine started.

SHA256 of the exact wrapper version I tested:

93d4235d5c79a3ce2aba22bbc80e049601ca0d5ca88a845f24afb67ed928231f

7. Important note

This is specifically a workaround for the current Divert + Hyperscan live-reload problem described in:

OPNsense core issue #10416

The automatic path deliberately avoids:
pkill -USR2 suricata
and performs a full restart when the active ruleset changes.

A full restart introduces a short inspection interruption while Suricata starts again, so anyone using this should evaluate that behavior for their own inline setup.

The wrapper above also checks OPNsense Divert listeners on:
*:8000

because that is the Divert configuration used on my system. Anyone with a substantially different setup should review the listener detection before copying it unchanged.

This is not intended to replace Suricata's normal live-reload mechanism permanently. Once the upstream Divert/Hyperscan issue is fixed and that fix is available in OPNsense, I intend to remove this workaround and return to the stock OPNsense rule-update path.

I hope this helps somebody else also.

This applies specifically to OPNsense Suricata IPS using Divert mode + Hyperscan. Do not use this workaround for normal Netmap/IDS installations unless you understand why you need it.