Hello,
I am running latest 26.1.10 under Unraid VM (QEMU) and a permanent issue that
when doing a reboot or shutdown it is getting stuck trying to kill Suricata forever.
root@OPNsense:~ # /usr/local/etc/rc.reboot
>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
crowdsec_firewall is not running.
Stopping crowdsec.
Waiting for PIDS: 22448.
lldpd not running? (check /var/run/lldpd.pid).
qemu_guest_agent not running? (check /var/run/qemu-ga.pid).
snmpd not running? (check /var/run/net_snmpd.pid).
Stopping suricata.
Waiting for PIDS: 26425
I had it sit for several minutes but still stuck.
I then permanently killed it manually by issuing a separate "kill -9 26425" which then let shutdown to continue.
root@OPNsense:~ # /usr/local/etc/rc.reboot
>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
crowdsec_firewall is not running.
crowdsec not running? (check /var/run/crowdsec_daemon.pid).
lldpd not running? (check /var/run/lldpd.pid).
qemu_guest_agent not running? (check /var/run/qemu-ga.pid).
snmpd not running? (check /var/run/net_snmpd.pid).
Stopping suricata.
Waiting for PIDS: 26425.
Stopping acme_http_challenge.
Waiting for PIDS: 16362.
Stopping flowd.
kill: 6470: No such process
kill: 7055: No such process
Stopping maltrailsensor.
Waiting for PIDS: 91290.
Stopping maltrailserver.
Waiting for PIDS: 88043.
Stopping apcupsd.
kill: 62174: No such process
Stopping flowd_aggregate...done
Stopping monit.
Waiting for PIDS: 85295.
crowdsec not running? (check /var/run/crowdsec_daemon.pid).
crowdsec_firewall is not running.
Stopping tailscaled.
Waiting for PIDS: 44920, 44920.
>>> Invoking stop script 'backup'
>>> Invoking backup script 'captiveportal'
>>> Invoking backup script 'netflow'
>>> Invoking backup script 'rrd'
>>> Invoking stop script 'config'
Shutdown NOW!
shutdown: [pid 90818]
*** FINAL System shutdown message from root@OPNsense.mrzaz.com ***
System going down IMMEDIATELY
*** FINAL System shutdown message from root@OPNsense.mrzaz.com ***
System going down IMMEDIATELY
This is what came in other session where i killed the process
root@OPNsense:~ # kill -9 26425
*** FINAL System shutdown message from root@OPNsense.mrzaz.com ***
System going down IMMEDIATELY
*** FINAL System shutdown message from root@OPNsense.mrzaz.com ***
System going down IMMEDIATELY
I have tried this several times at various times and get the same issue everytime. 100% failure.
I do have the "os-qemu-guest-agent" installed/running.
Does anyone else having same issue ?
Any idea of any workaround I could test ?
In which mode is suricata running? IDS, IPS (netmap or divert)?
Cheers,
Franco
https://forum.opnsense.org/index.php?topic=52191.msg269089;topicseen#msg269089
Double post or similar but separate problem?
Quote from: cookiemonster on June 25, 2026, 03:25:38 PMhttps://forum.opnsense.org/index.php?topic=52191.msg269089;topicseen#msg269089
Double post or similar but separate problem?
This issue is related to that issue as he mentions there two or three times ;)
Quote from: franco on June 25, 2026, 10:44:03 AMIn which mode is suricata running? IDS, IPS (netmap or divert)?
Cheers,
Franco
- Divert (IPS)
Pattern matcher = Aho-Corasick, "Ken Steele" variant.
//Dan Lundqvist
Quote from: cookiemonster on June 25, 2026, 03:25:38 PMhttps://forum.opnsense.org/index.php?topic=52191.msg269089;topicseen#msg269089
Double post or similar but separate problem?
Yes, I have closed that and keeping this only.
Sorry. Was not sure in which group it was best suited.
//Dan Lundqvist
Quote from: nero355 on June 25, 2026, 05:32:47 PMQuote from: cookiemonster on June 25, 2026, 03:25:38 PMhttps://forum.opnsense.org/index.php?topic=52191.msg269089;topicseen#msg269089
Double post or similar but separate problem?
This issue is related to that issue as he mentions there two or three times ;)
Hi, that other topic was really not about this issue and I only casually mentioned it as a side-effect.
But it could possible be the main root why it never shut down as it is hanging waiting for suricata PID to stop but never does unless i brutally kill it.
So question is how to proceed.
I will try what was proposed and we'll see what happens.
>Try running "/usr/local/etc/rc.d/suricata onestop" in the terminal and see what happens and then go and check what's in the "Services -> Intrusion Detection -> >Log File"
>
>/usr/local/etc/rc.d/suricata onestop
//Dan Lundqvist
I tried the proposed in other thread.
I get the following in the console:
root@OPNsense:~ # /usr/local/etc/rc.d/suricata onestop
Stopping suricata.
Waiting for PIDS: 71649.
root@OPNsense:~ #
and the following in Suricata Logfile:
2026-06-25T20:47:04 Notice suricata [100787] <Notice> -- (W-8000) Verdict: Accepted 0, Dropped 0
2026-06-25T20:47:04 Notice suricata [100787] <Notice> -- (W-8000) Treated: Pkts 0, Bytes 0, Errors 0
2026-06-25T20:47:04 Notice suricata [100786] <Notice> -- (W-8000) Verdict: Accepted 0, Dropped 0
2026-06-25T20:47:04 Notice suricata [100786] <Notice> -- (W-8000) Treated: Pkts 0, Bytes 0, Errors 0
2026-06-25T20:47:04 Notice suricata [100785] <Notice> -- (W-8000) Verdict: Accepted 0, Dropped 0
2026-06-25T20:47:04 Notice suricata [100785] <Notice> -- (W-8000) Treated: Pkts 0, Bytes 0, Errors 0
2026-06-25T20:47:03 Notice suricata [100642] <Notice> -- Signal Received. Stopping engine.
So when doing this, it shuts down but when it is done as the part of OpnSense shutdown it just hangs.
or at least it says that. It never passes the killing of PID for surcata and never continues with rest
of the shutdown procedure.
//Dan Lundqvist
Can you confirm this only happens with divert? It may be an open file descriptor / socket that the kernel doesn't yield.
Cheers,
Franco
Quote from: franco on June 25, 2026, 09:52:03 PMCan you confirm this only happens with divert? It may be an open file descriptor / socket that the kernel doesn't yield.
Cheers,
Franco
Feels like it happens intermittent now. I tried one more time from console and then it was able to kill all.
Will try one more time from GUI. Jupp. now that worked as well.
Must be something hanging that is now cleared somehow.
I will monitor this the comming days to see if it re-appers.
//Dan Lundqvist
Quote from: franco on June 25, 2026, 09:52:03 PMCan you confirm this only happens with divert? It may be an open file descriptor / socket that the kernel doesn't yield.
Cheers,
Franco
If it happens agen I will try to check that setting. I prefer to use Divert in the way that is handled.
I've looked at the code and it's unclear where Suricata would hang. Has to be in poll() or recvfrom() but both have timeouts and SIGINT/SIGTERM should be properly handled and seen by the application eventually within the span of a second.
Cheers,
Franco
Quote from: franco on June 26, 2026, 08:10:32 AMI've looked at the code and it's unclear where Suricata would hang. Has to be in poll() or recvfrom() but both have timeouts and SIGINT/SIGTERM should be properly handled and seen by the application eventually within the span of a second.
Cheers,
Franco
Hi Franco,
Thanks for the reply.
It is quite weird. When the issue was present i got the endless waiting for PID suricata that never end. I had to forcefully kill it for the rest of shutdown/reboot stuff to continue. Also when done from gui you didn't know why it was not shutting down or reboot.
I will keep checking if I stumble on it again.
If I do, do you have any commands to print to try to find what is going on?
Best regards
Dan Lundqvist
Stockholm, Sweden
Hi Franco,
I have again ended up with same issue.
I tried to reboot opnsense from GUI and nothing happens on any stdout.
I then checked if suricata process is running which it does.
root@OPNsense:~ # ps aux | sort -nr -k4 | head -20
root 36693 0.0 24.8 16206888 5196580 - Ss Thu21 13:33.69 /usr/local/bin/suricata -D -d 8000 -d 8000 -d 8000 --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
root 9539 0.0 16.0 3698000 3337860 - S Thu21 17:14.09 /usr/local/bin/python3 /usr/local/share/maltrail/sensor.py (python3.13)
root 57558 0.0 11.9 2655060 2485608 - S Thu21 64:17.87 /usr/local/bin/python3 /usr/local/share/maltrail/sensor.py (python3.13)
root 45012 1.0 2.7 1915168 568444 - S Thu21 62:42.00 /usr/local/bin/tailscaled -port 41641 -tun tailscale0 -statedir /var/db/tailscale
root 80262 0.0 0.6 360832 116516 - I Thu21 3:27.83 /usr/local/bin/python3 /usr/local/opnsense/scripts/unbound/logger.py (python3.13)
root@OPNsense:~ # more /var/run/suricata.pid
36693
It should have been killed by now as I have waited several minutes.
I tried again in console and gets stuck killing suricata.
The system will reboot. Do you want to proceed? [y/N]: y
>>> Invoking stop script 'beep'
>>> Invoking stop script 'freebsd'
crowdsec_firewall is not running.
crowdsec not running? (check /var/run/crowdsec_daemon.pid).
lldpd not running? (check /var/run/lldpd.pid).
qemu_guest_agent not running? (check /var/run/qemu-ga.pid).
snmpd not running? (check /var/run/net_snmpd.pid).
Stopping suricata.
Waiting for PIDS: 36693
Same issue as before...
Not sure if related but have some issues with swap space so have added a second 2GB swap file.
swap_pager: out of swap space
swp_pager_getswapspace(18): failed
When I managed to clear the situation earlier by doing "/usr/local/etc/rc.d/suricata onestop"
and then it worked OK after reboot ro do another reboot using console.
But now back to hanging suricata. :-|
In system Backend log i found:
2026-06-26T23:48:48 Error configd.py Timeout (120) executing : 'ids' restart
Will try once more with /usr/local/etc/rc.d/suricata onestop
Not even that one managed to kill suricata.
root@OPNsense:~ # /usr/local/etc/rc.d/suricata onestop
Stopping suricata.
Waiting for PIDS: 36693
"kill -9 36693" was the only one tough enough to kill it.
I will try to reconfigure from "Divert(IPS) to Netmap(IPS) to see if issue persists.
/Dan Lundqvist
Hi Franco,
I am facing the same issue. I discovered when the system tried to reboot on last update to Business 26.4.1.
My hardware is from Deciso: DEC 697
I had to connect into the console and kill suricata manually as it never rebooted for more than 10 minutes.
My suricata configuration is in divert mode. Since this divert mode became available, I have switched from IPS mode to divert mode as it makes more sense to inspect in suricata only what firewall is allowing, in my case, one specific rule, instead of inspecting all traffic.
I tried the same command showed as before, but the result was always the same. hanging when trying to stop suricata.
I didn't try changing the suricata mode back to IPS or IDS, but as far as I remember, I nave never experienced this hanging issue before. I have been using OpnSense for more than 3 years and this is the first time I encountered this hanging behavior. All previous updates was always smooth with no issues or hanging.
Regards,
Jorge
Quote from: Jorgek on June 29, 2026, 08:23:06 AMHi Franco,
I am facing the same issue. I discovered when the system tried to reboot on last update to Business 26.4.1.
My hardware is from Deciso: DEC 697
I had to connect into the console and kill suricata manually as it never rebooted for more than 10 minutes.
My suricata configuration is in divert mode. Since this divert mode became available, I have switched from IPS mode to divert mode as it makes more sense to inspect in suricata only what firewall is allowing, in my case, one specific rule, instead of inspecting all traffic.
I tried the same command showed as before, but the result was always the same. hanging when trying to stop suricata.
I didn't try changing the suricata mode back to IPS or IDS, but as far as I remember, I nave never experienced this hanging issue before. I have been using OpnSense for more than 3 years and this is the first time I encountered this hanging behavior. All previous updates was always smooth with no issues or hanging.
Regards,
Jorge
Hi Jorge,
Then at least I am not alone in this. 🙂
Due to HW constraints in my old opnsense machine I did not use Suricata that much but has now enabled it more and that's when I discovered it.
It always hanged when trying to shutdown. Only thing powerfully enough to kill it was -9.
Dan Lundqvist
I have now changed from Divert (IPS) to Netmap (IDS) and let it run for 24-36h and now tried a normal reboot and at least this time it rebooted normally.
Only took a few seconds for suricata PID to stop and continue with rest of the shutdown/reboot.
I will keep this under wrap and test it again in a few days.
If it now is Divert setting that causes it, we need to try to find the culprit.
I will try to revert to Divert (IPS) and see if I could reproduce and then use a bunch of hopefully good commands to debug.
//Dan Lundqvist
Hello,
I had it running for 1-2 days using "Netmap (IPS)" and did a few manual reboots and it shuts down/reboot OK.
I then reconfig it to "Divert (IPS)" and then after a while I got the same hard lock.
I did a lot of debug printouts that I could send if someone is interested.
I tried first "kill -TERM 73697" but that did nothing. process still hanging.
I then did a "kill -9 73697" and then it continued all the way to reboot and then up again.
Seems like it is happening in relative short time on "Divert (IPS)".
//Dan Lundqvist
Stockholm, Sweden
Quote from: franco on June 25, 2026, 09:52:03 PMCan you confirm this only happens with divert? It may be an open file descriptor / socket that the kernel doesn't yield.
Cheers,
Franco
Hi Franco,
I have now tested with Divert (IPS), then Netmap (IPS) and then back to Divert (IPS) again.
Result is that with "Netmap (IPS)" I do not get this hangings and it could run for hours and
every time I tried a reboot it shutdown clean and restarted as normal.
But when I reverted back to "Divert (IPS) and let it run for a few hours and then try
a reboot it hangs waiting for suricata PID to die. I then tried "kill TERM <pid>"
but did not help. I then had to go for the big gun with "term -9 <pid>" and then
it continued the shutdown and rebooted OK.
I do have some trace printouts that I could send if someone wants to review ?
//Dan Lundqvist
I'm not sure what the best thing to look at is, but let's start with the obvious during a hang vs. what I can see here during normal operation:
# sockstat | grep suricata
root suricata 92847 3 dgram ?? -> /var/run/log
root suricata 92847 6 div4 *:8000 *:*
# ps auxwww | grep suricata
root 92847 0.6 1.3 155104 107252 - Ss 19:50 0:01.29 /usr/local/bin/suricata -D -d 8000 --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
root 70011 0.0 0.0 14088 2620 1 S+ 19:51 0:00.00 grep suricata
# fstat | grep suricata
root suricata 92847 text / 6012 -rwxr-xr-x 11163768 r
root suricata 92847 wd / 34 drwxr-xr-x 28 r
root suricata 92847 root / 34 drwxr-xr-x 28 r
root suricata 92847 0 /dev 20 crw-rw-rw- null rw
root suricata 92847 1 /dev 20 crw-rw-rw- null rw
root suricata 92847 2 /dev 20 crw-rw-rw- null rw
root suricata 92847 3* local dgram fffff800283498c0 <-> fffff800282b8280
root suricata 92847 4 /var/log 2246 -rwx------ 280764251 w
root suricata 92847 5 /var/log 5591 -rwx------ 1403297 w
root suricata 92847 6* divert raw 0 0
Cheers,
Franco
If you need a temporary solution, I have a suggestion. Add a timeout in the Suricata service script:
sed -i '' 's|command:/usr/local/etc/rc.d/suricata stop|command:/usr/local/etc/rc.d/suricata stop \|\| (sleep 10 \&\& killall -9 suricata)|' /usr/local/opnsense/service/conf/actions.d/actions_ids.conf
Quote from: franco on July 01, 2026, 07:52:16 PMI'm not sure what the best thing to look at is, but let's start with the obvious during a hang vs. what I can see here during normal operation:
# sockstat | grep suricata
root suricata 92847 3 dgram ?? -> /var/run/log
root suricata 92847 6 div4 *:8000 *:*
# ps auxwww | grep suricata
root 92847 0.6 1.3 155104 107252 - Ss 19:50 0:01.29 /usr/local/bin/suricata -D -d 8000 --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
root 70011 0.0 0.0 14088 2620 1 S+ 19:51 0:00.00 grep suricata
# fstat | grep suricata
root suricata 92847 text / 6012 -rwxr-xr-x 11163768 r
root suricata 92847 wd / 34 drwxr-xr-x 28 r
root suricata 92847 root / 34 drwxr-xr-x 28 r
root suricata 92847 0 /dev 20 crw-rw-rw- null rw
root suricata 92847 1 /dev 20 crw-rw-rw- null rw
root suricata 92847 2 /dev 20 crw-rw-rw- null rw
root suricata 92847 3* local dgram fffff800283498c0 <-> fffff800282b8280
root suricata 92847 4 /var/log 2246 -rwx------ 280764251 w
root suricata 92847 5 /var/log 5591 -rwx------ 1403297 w
root suricata 92847 6* divert raw 0 0
Cheers,
Franco
Hi Franc, see PM.
//Danne
Quote from: wincent on July 02, 2026, 03:41:51 AMIf you need a temporary solution, I have a suggestion. Add a timeout in the Suricata service script:
sed -i '' 's|command:/usr/local/etc/rc.d/suricata stop|command:/usr/local/etc/rc.d/suricata stop \|\| (sleep 10 \&\& killall -9 suricata)|' /usr/local/opnsense/service/conf/actions.d/actions_ids.conf
Thanks, I will check that. But best would be solving it more permanent. But will save yours anyway during meantime.
//Danne
Hello,
this seems to be the same issue i faced here (https://forum.opnsense.org/index.php?topic=52052.0)
I have suricata running in ips mode with divert.
so i updated to 26.1.11 and the issue is here
here is what i have pre update:
red@cerberus:~ $ sudo sockstat | grep suricata
root suricata 29596 3 dgram -> /var/run/log
root suricata 29596 6 div4 *:8000 *:*
fred@cerberus:~ $
fred@cerberus:~ $
fred@cerberus:~ $ sudo ps auxwww | grep suricata
root 29596 0.1 23.8 2723192 1922216 - Ss 20Jun26 96:13.75 /usr/local/bin/suricata -D -d 8000 --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
fred 19042 0.0 0.0 13744 2032 0 S+ 10:40 0:00.00 grep suricata
fred@cerberus:~ $
fred@cerberus:~ $
fred@cerberus:~ $ sudo fstat | grep suricata
root suricata 29596 text / 248115 -rwxr-xr-x 11994960 r
root suricata 29596 wd / 34 drwxr-xr-x 28 r
root suricata 29596 root / 34 drwxr-xr-x 28 r
root suricata 29596 0 /dev 20 crw-rw-rw- null rw
root suricata 29596 1 /dev 20 crw-rw-rw- null rw
root suricata 29596 2 /dev 20 crw-rw-rw- null rw
root suricata 29596 3* local dgram fffff8001bb27640 <-> fffff8001bce2dc0
root suricata 29596 4 /var/log 27980 -rw-r----- 0 w
root suricata 29596 5 - 476 -rw-r----- 6318542 w
root suricata 29596 6* divert raw 0 0
and what i have when suricata is stucked:
fred@cerberus:~ $ sudo sockstat | grep suricata
root suricata 29596 3 dgram (not connected)
root suricata 29596 6 div4 *:8000 *:*
fred@cerberus:~ $
fred@cerberus:~ $ sudo ps auxwww | grep suricata
root 29596 0.1 23.8 2723192 1922216 - Ss 20Jun26 96:15.42 /usr/local/bin/suricata -D -d 8000 --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
root 44586 0.0 0.0 14312 2888 - I 10:44 0:00.01 /bin/sh /usr/local/etc/rc.d/suricata stop
fred 67953 0.0 0.0 13744 2336 0 S+ 10:46 0:00.00 grep suricata
fred@cerberus:~ $
fred@cerberus:~ $ sudo fstat | grep suricata
root suricata 29596 text / 248115 -rwxr-xr-x 11994960 r
root suricata 29596 wd / 34 drwxr-xr-x 28 r
root suricata 29596 root / 34 drwxr-xr-x 28 r
root suricata 29596 0 /dev 20 crw-rw-rw- null rw
root suricata 29596 1 /dev 20 crw-rw-rw- null rw
root suricata 29596 2 /dev 20 crw-rw-rw- null rw
root suricata 29596 3* local dgram fffff8001bb27640
root suricata 29596 4 /var/log 27980 -rw-r----- 0 w
root suricata 29596 5 - 476 -rw-r----- 6320460 w
root suricata 29596 6* divert raw 0 0
when i killed pid 44586 opnsense was able to reboot and. the upgrade is ok.
Thanks for the debug info so far.
I'm really unsure what the problem is here, but I've added a little patch to diagnose if the SIGINT is not properly handled in the divert packet loop.
https://github.com/opnsense/ports/commit/7883cddc3
# opnsense-revert -z suricata
(it should install 8.0.5_2 and you need to restart suricata to activate)
Cheers,
Franco
Quote from: franco on July 02, 2026, 03:13:31 PMopnsense-revert -z suricata
I have now reverted to 8.0.5_2 and killed and restarted suricata.
Running "Deivert (IPS)" mode. (Where I have seen the hanging.)
How will I get any info out if it hangs again during shutdown ?
//Dan Lundqvist
i installed the patch. What should i look for ? the same commands ?
Just let me know if suricata is able to exit now or not. I've put an additional exit condition between the poll and recvfrom calls that are obviously being looped over. The thread-based exit vars are never checked (neither for netmap so I don't understand why only ipfw is affected but we can test more later).
Cheers,
Franco
👍
Danne
I updated to 26.1.11_5 but no reboot was necessary.
I manually rebooted without issue. So i guess there was no problem to shut down suricata.
Ok, let's give it a few days. Thanks for testing so far!
Cheers,
Franco
the least i can do given the effort you put in it.
Hi franc and others,
Unfortunatly it hang with 8.0.5_2 suricata and only kill -9 kills it.
suricata 8.0.5_2 59.3MiB unknown-repository GPLv2 High Performance Network IDS, IPS and Security Monitoring engine
root@OPNsense:~ # uptime
5:21PM up 2 days, 8:40, 2 users, load averages: 0.72, 0.49, 0.54
Running "Divert (UPS)".
Any other ideas what to modify and try again ? Maybe another "black" fix in suricata ?
//Dan Lundqvist
//Dan Lundqvist
Just for info.
Upgraded to latest but it still hanged as before not able to shut down unless kill -9 <suricata process>
It is still running suricata 8.0.5_2 59.3MiB unknown-repository GPLv2 High Performance Network IDS, IPS and Security Monitoring engine
//Dan Lundqvist
Okay, thanks. All of this feels like it doesn't propagate the exit. Next step would be for me to try and reproduce the behaviour but so far no luck.
How much traffic are you pushing into suricata?
Cheers,
Franco
I have some dynamic data where most of the time it is limited but at certain times I run backups from Synology to external systems
and at other times I have a friend who backups to my storage.
Unfortunatley I had to revert to non-VM now as I hade a lot of other issues caused me to pause the Unraid and use same machine
as standalone OpnSense. I will check if I still get same issue.
I do have 1000/1000 connection and at times both runs much UL and sometimes much DL to system that has similar speeds as me.
//Dan Lundqvist
No news here, but I was just thinking 26.7 may be a good data point to add. No rush, but we could just have been missing a kernel-side commit since we couldn't find an issue with userspace.
Cheers,
Franco
Quote from: franco on July 15, 2026, 11:03:48 AMNo news here, but I was just thinking 26.7 may be a good data point to add. No rush, but we could just have been missing a kernel-side commit since we couldn't find an issue with userspace.
Cheers,
Franco
Thanks. I will try to monitor this in my system if I find the same issue.
Also has some other issues with IDS (libpcap) as well as it feels like the data does not reach suricata.
But that is another ballgame I will touch upon later. I would prefer to use the Divert as this will
remove a lot of the load by only sending through stuff that is not catched by the normal firewall.
However, i will try to see if I could reproduce again.
I think I need to get att better NIC with more than 1/1 channels as I get a lot of
netmap: tx ring full messages
Best regards
Dan Lundqvist