"
Hi,
I am running OPNsense:
OPNsense 26.7.2_2-amd64
FreeBSD 15.1-RELEASE-p2
OpenSSL 3.5.7
With Suricata IPS in Divert mode and Hyperscan.
There is currently an open OPNsense issue describing an intermittent deadlock during a live Suricata rule reload with SIGUSR2 in this configuration:
https://github.com/opnsense/core/issues/10416
The stock scheduled IDS rule update currently follows this path:
The workaround described in the issue is to update/install the rules and then perform a full Suricata restart instead of entering the problematic live-reload path.
I wanted to keep automatic rule updates, but add some safeguards around that workaround.
The wrapper below prevents concurrent runs with lockf, uses OPNsense's own rule updater and installer, backs up the active ruleset, skips an unnecessary restart when the active rules did not change, validates the new rules before restarting, performs a full restart without SIGUSR2, checks whether the Divert listeners return, and restores the previous active ruleset if validation/startup fails.
This is a temporary workaround, not an upstream fix.
1. Create the update wrapper
Create:
/usr/local/sbin/suricata-safe-update
contents:
Set ownership and permissions:
Check shell syntax:
Optional check that the wrapper itself does not contain a SIGUSR2, pkill, or kill command:
No output is expected.
2. Create a configd action
Create:
/usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.conf
Contents:
Set permissions:
Reload configd:
Check that the action was registered:
Expected:
suricatasafe update [ Suricata safe rule update (full restart, no SIGUSR2) ]
3. Test manually before changing Cron
Check the current Suricata PID:
Check the current Suricata sockets:
For the actual test I recommend using detached configctl:
A synchronous call may exceed the configctl client timeout because Hyperscan validation and startup can take several minutes.
Monitor the updater and Suricata:
Check the sockets:
After the update has finished, check the PID again:
Check whether the new engine completed startup:
A successful start should contain something similar to:
Threads created -> W: 8 FM: 1 FR: 1 Engine started.
The number of workers/listeners depends on the system configuration.
4. Replace the normal scheduled IDS update
Go to:
System → Settings → Cron
Disable the existing:
ids rule updates
I kept the original job present but disabled, rather than deleting it, so reverting is easy.
Then create a new Cron job using:
Suricata safe rule update (full restart, no SIGUSR2)
For example, daily at 02:00:
OPNsense should generate an effective cron entry similar to:
Verify the effective cron:
The old active command:
should be gone.
The new active command should be:
5. Why detached configctl -d?
On my system a real rule update plus Hyperscan validation and full Suricata restart takes longer than a synchronous configctl client waits.
During testing, a synchronous invocation produced:
The configd action itself continued running and completed successfully.
For scheduled execution I therefore use:
The lockf around the wrapper prevents two update jobs from running concurrently.
6. Tested result
Tested on:
A real rule update resulted in a full Suricata restart.
During startup the new Suricata process spent roughly two minutes at high CPU while compiling/initializing the rules.
Afterwards all eight Divert listeners on my system returned:
The log then reported:
Threads created -> W: 8 FM: 1 FR: 1 Engine started.
SHA256 of the exact wrapper version I tested:
7. Important note
This is specifically a workaround for the current Divert + Hyperscan live-reload problem described in:
OPNsense core issue #10416
The automatic path deliberately avoids:
and performs a full restart when the active ruleset changes.
A full restart introduces a short inspection interruption while Suricata starts again, so anyone using this should evaluate that behavior for their own inline setup.
The wrapper above also checks OPNsense Divert listeners on:
*:8000
because that is the Divert configuration used on my system. Anyone with a substantially different setup should review the listener detection before copying it unchanged.
This is not intended to replace Suricata's normal live-reload mechanism permanently. Once the upstream Divert/Hyperscan issue is fixed and that fix is available in OPNsense, I intend to remove this workaround and return to the stock OPNsense rule-update path.
I hope this helps somebody else also.
This applies specifically to OPNsense Suricata IPS using Divert mode + Hyperscan. Do not use this workaround for normal Netmap/IDS installations unless you understand why you need it.
I am running OPNsense:
OPNsense 26.7.2_2-amd64
FreeBSD 15.1-RELEASE-p2
OpenSSL 3.5.7
With Suricata IPS in Divert mode and Hyperscan.
There is currently an open OPNsense issue describing an intermittent deadlock during a live Suricata rule reload with SIGUSR2 in this configuration:
https://github.com/opnsense/core/issues/10416
The stock scheduled IDS rule update currently follows this path:
| rule-updater.py installRules.py pkill -USR2 suricata |
The workaround described in the issue is to update/install the rules and then perform a full Suricata restart instead of entering the problematic live-reload path.
I wanted to keep automatic rule updates, but add some safeguards around that workaround.
The wrapper below prevents concurrent runs with lockf, uses OPNsense's own rule updater and installer, backs up the active ruleset, skips an unnecessary restart when the active rules did not change, validates the new rules before restarting, performs a full restart without SIGUSR2, checks whether the Divert listeners return, and restores the previous active ruleset if validation/startup fails.
This is a temporary workaround, not an upstream fix.
1. Create the update wrapper
Create:
/usr/local/sbin/suricata-safe-update
contents:
Code Select
#!/bin/sh
set -u
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
export PATH
TAG="suricata-safe-update"
UPDATER="/usr/local/opnsense/scripts/suricata/rule-updater.py"
INSTALLER="/usr/local/opnsense/scripts/suricata/installRules.py"
SURICATA="/usr/local/bin/suricata"
CONFIG="/usr/local/etc/suricata/suricata.yaml"
RC="/usr/local/etc/rc.d/suricata"
ACTIVE_DIR="/usr/local/etc/suricata/opnsense.rules"
ACTIVE_YAML="/usr/local/etc/suricata/installed_rules.yaml"
BACKUP_DIR=""
logmsg()
{
logger -t "$TAG" "$*"
echo "$TAG: $*"
}
active_hash()
{
{
if [ -d "$ACTIVE_DIR" ]; then
find "$ACTIVE_DIR" -type f -name '*.rules' -print 2>/dev/null |
sort |
while IFS= read -r f; do
printf '%s ' "$f"
sha256 -q "$f"
done
fi
if [ -f "$ACTIVE_YAML" ]; then
printf '%s ' "$ACTIVE_YAML"
sha256 -q "$ACTIVE_YAML"
fi
} | sha256 -q
}
divert_count()
{
sockstat 2>/dev/null |
awk '$2 == "suricata" && $5 == "div4" && $6 == "*:8000" { n++ }
END { print n + 0 }'
}
restore_active()
{
logmsg "restoring previous active ruleset"
rm -rf "$ACTIVE_DIR"
if [ -d "$BACKUP_DIR/opnsense.rules" ]; then
cp -a "$BACKUP_DIR/opnsense.rules" "$ACTIVE_DIR"
fi
if [ -f "$BACKUP_DIR/installed_rules.yaml" ]; then
cp -p "$BACKUP_DIR/installed_rules.yaml" "$ACTIVE_YAML"
else
rm -f "$ACTIVE_YAML"
fi
}
cleanup()
{
if [ -n "${BACKUP_DIR:-}" ] && [ -d "$BACKUP_DIR" ]; then
rm -rf "$BACKUP_DIR"
fi
}
trap cleanup EXIT HUP INT TERM
for f in "$UPDATER" "$INSTALLER" "$SURICATA" "$CONFIG" "$RC"; do
if [ ! -e "$f" ]; then
logmsg "ABORT: required file missing: $f"
exit 10
fi
done
if ! "$RC" status >/dev/null 2>&1; then
logmsg "ABORT: Suricata is not running"
exit 11
fi
OLD_DIVERT=$(divert_count)
if [ "$OLD_DIVERT" -lt 1 ]; then
logmsg "ABORT: no active Suricata Divert listeners found"
exit 12
fi
OLD_PID=$(cat /var/run/suricata.pid 2>/dev/null || true)
BEFORE=$(active_hash)
logmsg "starting rule update; current PID=$OLD_PID divert_listeners=$OLD_DIVERT"
"$UPDATER"
rc=$?
if [ "$rc" -ne 0 ]; then
logmsg "FAILED: rule-updater.py rc=$rc"
exit "$rc"
fi
BACKUP_DIR=$(mktemp -d /root/suricata-safe-update.XXXXXX)
if [ -d "$ACTIVE_DIR" ]; then
cp -a "$ACTIVE_DIR" "$BACKUP_DIR/opnsense.rules"
fi
if [ -f "$ACTIVE_YAML" ]; then
cp -p "$ACTIVE_YAML" "$BACKUP_DIR/installed_rules.yaml"
fi
logmsg "installing updated rules"
"$INSTALLER"
rc=$?
if [ "$rc" -ne 0 ]; then
logmsg "FAILED: installRules.py rc=$rc"
restore_active
exit "$rc"
fi
AFTER=$(active_hash)
if [ "$BEFORE" = "$AFTER" ]; then
logmsg "no active rule changes; restart skipped"
exit 0
fi
logmsg "active rules changed; validating new ruleset"
"$SURICATA" -T --init-errors-fatal -c "$CONFIG"
rc=$?
if [ "$rc" -ne 0 ]; then
logmsg "FAILED: Suricata validation rc=$rc; restart NOT performed"
restore_active
exit "$rc"
fi
logmsg "validation passed; performing full Suricata restart"
"$RC" restart
rc=$?
if [ "$rc" -ne 0 ]; then
logmsg "FAILED: restart rc=$rc; restoring previous ruleset"
restore_active
"$RC" restart
exit 30
fi
i=0
while [ "$i" -lt 36 ]; do
sleep 5
NEW_DIVERT=$(divert_count)
if "$RC" status >/dev/null 2>&1 &&
[ "$NEW_DIVERT" -eq "$OLD_DIVERT" ]; then
NEW_PID=$(cat /var/run/suricata.pid 2>/dev/null || true)
logmsg "SUCCESS: PID=$NEW_PID divert_listeners=$NEW_DIVERT"
exit 0
fi
i=$((i + 1))
done
logmsg "FAILED: Divert listeners did not recover within 180 seconds"
logmsg "restoring previous ruleset and restarting"
restore_active
"$RC" restart
rc=$?
if [ "$rc" -ne 0 ]; then
logmsg "CRITICAL: rollback restart failed rc=$rc"
exit 40
fi
logmsg "ROLLBACK completed; manual verification required"
exit 41Set ownership and permissions:
Code Select
chown root:wheel /usr/local/sbin/suricata-safe-update
chmod 0700 /usr/local/sbin/suricata-safe-updateCheck shell syntax:
Code Select
/bin/sh -n /usr/local/sbin/suricata-safe-updateOptional check that the wrapper itself does not contain a SIGUSR2, pkill, or kill command:
Code Select
grep -nEi 'USR2|pkill|kill ' /usr/local/sbin/suricata-safe-updateNo output is expected.
2. Create a configd action
Create:
/usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.conf
Contents:
Code Select
[update]
command:/usr/bin/lockf -k -t 0 /var/run/suricata-safe-update.lock /usr/local/sbin/suricata-safe-update
parameters:
type:script
message:safely update Suricata rules with full restart when needed
description:Suricata safe rule update (full restart, no SIGUSR2)Set permissions:
Code Select
chown root:wheel /usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.conf
chmod 0644 /usr/local/opnsense/service/conf/actions.d/actions_suricatasafe.confReload configd:
Code Select
service configd restartCheck that the action was registered:
Code Select
configctl configd actions | grep -i suricatasafeExpected:
suricatasafe update [ Suricata safe rule update (full restart, no SIGUSR2) ]
3. Test manually before changing Cron
Check the current Suricata PID:
Code Select
cat /var/run/suricata.pidCheck the current Suricata sockets:
Code Select
sockstat | grep '[s]uricata'For the actual test I recommend using detached configctl:
Code Select
/usr/local/sbin/configctl -d -- suricatasafe updateA synchronous call may exceed the configctl client timeout because Hyperscan validation and startup can take several minutes.
Monitor the updater and Suricata:
Code Select
ps -axww -o pid,ppid,state,etime,%cpu,%mem,rss,vsz,command | grep -E 'suricata-safe-update|suricata -T|/usr/local/bin/suricata' | grep -v grepCheck the sockets:
Code Select
sockstat | grep '[s]uricata'After the update has finished, check the PID again:
Code Select
cat /var/run/suricata.pidCheck whether the new engine completed startup:
Code Select
/bin/sh -c 'PID=$(cat /var/run/suricata.pid); grep "suricata $PID" /var/log/suricata/suricata_*.log | grep -E "Engine started|<Error>|<Warning>" | tail -30'A successful start should contain something similar to:
Threads created -> W: 8 FM: 1 FR: 1 Engine started.
The number of workers/listeners depends on the system configuration.
4. Replace the normal scheduled IDS update
Go to:
System → Settings → Cron
Disable the existing:
ids rule updates
I kept the original job present but disabled, rather than deleting it, so reverting is easy.
Then create a new Cron job using:
Suricata safe rule update (full restart, no SIGUSR2)
For example, daily at 02:00:
| Minutes: 0 Hours: 2 Days: * Months: * Weekdays: * |
OPNsense should generate an effective cron entry similar to:
Code Select
0 2 * * * /usr/local/sbin/configctl -d -- suricatasafe updateVerify the effective cron:
Code Select
grep -nE 'ids update|suricatasafe update' /var/cron/tabs/nobodyThe old active command:
Code Select
/usr/local/sbin/configctl -d -- ids updateshould be gone.
The new active command should be:
Code Select
/usr/local/sbin/configctl -d -- suricatasafe update5. Why detached configctl -d?
On my system a real rule update plus Hyperscan validation and full Suricata restart takes longer than a synchronous configctl client waits.
During testing, a synchronous invocation produced:
| error in configd communication TimeoutError: timed out |
The configd action itself continued running and completed successfully.
For scheduled execution I therefore use:
Code Select
/usr/local/sbin/configctl -d -- suricatasafe updateThe lockf around the wrapper prevents two update jobs from running concurrently.
6. Tested result
Tested on:
| OPNsense: 26.7.2_2 Suricata: 8.0.6 IPS mode: Divert Pattern matcher: Hyperscan |
A real rule update resulted in a full Suricata restart.
During startup the new Suricata process spent roughly two minutes at high CPU while compiling/initializing the rules.
Afterwards all eight Divert listeners on my system returned:
| div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 div4 *:8000 |
The log then reported:
Threads created -> W: 8 FM: 1 FR: 1 Engine started.
SHA256 of the exact wrapper version I tested:
| 93d4235d5c79a3ce2aba22bbc80e049601ca0d5ca88a845f24afb67ed928231f |
7. Important note
This is specifically a workaround for the current Divert + Hyperscan live-reload problem described in:
OPNsense core issue #10416
The automatic path deliberately avoids:
Code Select
pkill -USR2 suricataand performs a full restart when the active ruleset changes.
A full restart introduces a short inspection interruption while Suricata starts again, so anyone using this should evaluate that behavior for their own inline setup.
The wrapper above also checks OPNsense Divert listeners on:
*:8000
because that is the Divert configuration used on my system. Anyone with a substantially different setup should review the listener detection before copying it unchanged.
This is not intended to replace Suricata's normal live-reload mechanism permanently. Once the upstream Divert/Hyperscan issue is fixed and that fix is available in OPNsense, I intend to remove this workaround and return to the stock OPNsense rule-update path.
I hope this helps somebody else also.
This applies specifically to OPNsense Suricata IPS using Divert mode + Hyperscan. Do not use this workaround for normal Netmap/IDS installations unless you understand why you need it.
