Suricata not exiting on stop in Divert (IPS) mode

[jordanwmcdonald]

The old "suricata in IPS mode won't stop cleanly, GUI says stopped but the process is still running, needs a manual kill" behavior is well known from the netmap days (opnsense/core#3868, #4257, #710; and here on the forum, "Hanging at reboot: stopping suricata" back on 21.7). I ran into what looks like the same class of thing in the newer Divert mode, and in my case it hung a reboot badly enough to be worth flagging. I put the reboot details in opnsense/core#10567.

Setup here: 27.1.a devel, Divert (IPS), 12 divert threads (-d 8000 x12), a single narrow divert-to flow that sees very little traffic. On stop, suricata caught SIGTERM (procstat -i: TERM --C) and started shutting down but never exited. rc.d/suricata stop then sat in pwait for over 90 minutes. procstat showed the divert worker threads parked in sbwait/select on the divert socket, plus a few in umtx waits. My best guess is the workers are blocked reading the divert socket with nothing to wake them on this quiet flow, so the shutdown never completes, but it's only a guess.

Is the divert-mode stop hang tracked separately from the old netmap lockups, or is it the same known "kill it and move on" behavior? I can characterize it further if that helps (traffic during the stop, thread count, PCAP vs divert). Otherwise I'll keep a kill -9 in my stop path and leave it there.

[franco]

Hey,

Larger thread here https://forum.opnsense.org/index.php?topic=52203.0 testing was inconclusive so far.

Doesn't seem to affect netmap so it's same same but different.

If you can find out more that would be helpful.


Cheers,
Franco

[jordanwmcdonald]

Thanks for the pointer, I hadn't found that thread. I tore the divert setup down last week and picked up two things on the way out that might be worth something, though I'm not sure how much.

The one I didn't expect is that log rotation had quietly stopped working. newsyslog rotates suricata's logs against the pid file with signal 1, which is what the IDS template ships. Suricata seems to have honored the first rotation after it started and then none of the later ones. Its descriptors stayed on the rotated inodes while the new eve.json and stats.log both sat at zero bytes. I'm giving inodes rather than filenames because the chain has shifted since, but as of Jul 29, fd 4 was on inode 1066, which was eve.json.0 by then, while the live eve.json (inode 1453) was empty, and fd 5 was on inode 1031 with the live stats.log also empty. Four consecutive daily stats.log.N files at exactly 0 bytes was what finally made me look. Restarting cleared it.

I should be honest that the "first one worked, later ones didn't" part is me reading file sizes, not a log line saying so. The file covering engine start to the first rotation had 176,607 bytes in 7h42m, and inode 1031 then reached 3,048,418 bytes over the next 5d12h, so roughly 22.9 and 23.1 KB/h. I did check it was one process throughout, two engine starts in the logs and nothing in between. But it could have another explanation I'm not seeing.

What I can't work out is whether this is even related. It's happening well before anything tries to stop the process, so it may just be a separate thing that's wrong in divert mode. You mentioned suspecting a descriptor the kernel isn't yielding, and I genuinely don't know whether a log fd that never gets reopened points in the same direction or somewhere else entirely. That's your call rather than mine.

On the stop hang itself, one thing I can rule out is traffic. I cleared divert-to from the pf rule, applied, checked pfctl -sr had none left, and only then sent SIGTERM, so nothing was in flight and nothing was being diverted. It still sat for 20 seconds and I ended up using kill -9. So the quiet-flow idea from my first post doesn't hold up, an idle socket is enough on its own.

Also, saying it "ignores SIGTERM" was sloppy of me. It does get the signal and does start shutting down, procstat showed TERM --C and the log line in the other thread looks the same. It just doesn't finish.

I never looked at the divert worker code, so the sbwait/select states I quoted last time are only what procstat showed.

The box is back on PCAP now, so I can't dig any further without putting it into divert again and losing the coverage. Posting in case any of it is useful. Entirely possible it's all one thing you already know about.

(For transparency: I used Claude Opus 5 to help work through the diagnostics and to draft this post. Everything above is from my own box and I've checked the numbers myself.)