"
Thanks for the pointer, I hadn't found that thread. I tore the divert setup down last week and picked up two things on the way out that might be worth something, though I'm not sure how much.
The one I didn't expect is that log rotation had quietly stopped working. newsyslog rotates suricata's logs against the pid file with signal 1, which is what the IDS template ships. Suricata seems to have honored the first rotation after it started and then none of the later ones. Its descriptors stayed on the rotated inodes while the new eve.json and stats.log both sat at zero bytes. I'm giving inodes rather than filenames because the chain has shifted since, but as of Jul 29, fd 4 was on inode 1066, which was eve.json.0 by then, while the live eve.json (inode 1453) was empty, and fd 5 was on inode 1031 with the live stats.log also empty. Four consecutive daily stats.log.N files at exactly 0 bytes was what finally made me look. Restarting cleared it.
I should be honest that the "first one worked, later ones didn't" part is me reading file sizes, not a log line saying so. The file covering engine start to the first rotation had 176,607 bytes in 7h42m, and inode 1031 then reached 3,048,418 bytes over the next 5d12h, so roughly 22.9 and 23.1 KB/h. I did check it was one process throughout, two engine starts in the logs and nothing in between. But it could have another explanation I'm not seeing.
What I can't work out is whether this is even related. It's happening well before anything tries to stop the process, so it may just be a separate thing that's wrong in divert mode. You mentioned suspecting a descriptor the kernel isn't yielding, and I genuinely don't know whether a log fd that never gets reopened points in the same direction or somewhere else entirely. That's your call rather than mine.
On the stop hang itself, one thing I can rule out is traffic. I cleared divert-to from the pf rule, applied, checked pfctl -sr had none left, and only then sent SIGTERM, so nothing was in flight and nothing was being diverted. It still sat for 20 seconds and I ended up using kill -9. So the quiet-flow idea from my first post doesn't hold up, an idle socket is enough on its own.
Also, saying it "ignores SIGTERM" was sloppy of me. It does get the signal and does start shutting down, procstat showed TERM --C and the log line in the other thread looks the same. It just doesn't finish.
I never looked at the divert worker code, so the sbwait/select states I quoted last time are only what procstat showed.
The box is back on PCAP now, so I can't dig any further without putting it into divert again and losing the coverage. Posting in case any of it is useful. Entirely possible it's all one thing you already know about.
(For transparency: I used Claude Opus 5 to help work through the diagnostics and to draft this post. Everything above is from my own box and I've checked the numbers myself.)
The one I didn't expect is that log rotation had quietly stopped working. newsyslog rotates suricata's logs against the pid file with signal 1, which is what the IDS template ships. Suricata seems to have honored the first rotation after it started and then none of the later ones. Its descriptors stayed on the rotated inodes while the new eve.json and stats.log both sat at zero bytes. I'm giving inodes rather than filenames because the chain has shifted since, but as of Jul 29, fd 4 was on inode 1066, which was eve.json.0 by then, while the live eve.json (inode 1453) was empty, and fd 5 was on inode 1031 with the live stats.log also empty. Four consecutive daily stats.log.N files at exactly 0 bytes was what finally made me look. Restarting cleared it.
I should be honest that the "first one worked, later ones didn't" part is me reading file sizes, not a log line saying so. The file covering engine start to the first rotation had 176,607 bytes in 7h42m, and inode 1031 then reached 3,048,418 bytes over the next 5d12h, so roughly 22.9 and 23.1 KB/h. I did check it was one process throughout, two engine starts in the logs and nothing in between. But it could have another explanation I'm not seeing.
What I can't work out is whether this is even related. It's happening well before anything tries to stop the process, so it may just be a separate thing that's wrong in divert mode. You mentioned suspecting a descriptor the kernel isn't yielding, and I genuinely don't know whether a log fd that never gets reopened points in the same direction or somewhere else entirely. That's your call rather than mine.
On the stop hang itself, one thing I can rule out is traffic. I cleared divert-to from the pf rule, applied, checked pfctl -sr had none left, and only then sent SIGTERM, so nothing was in flight and nothing was being diverted. It still sat for 20 seconds and I ended up using kill -9. So the quiet-flow idea from my first post doesn't hold up, an idle socket is enough on its own.
Also, saying it "ignores SIGTERM" was sloppy of me. It does get the signal and does start shutting down, procstat showed TERM --C and the log line in the other thread looks the same. It just doesn't finish.
I never looked at the divert worker code, so the sbwait/select states I quoted last time are only what procstat showed.
The box is back on PCAP now, so I can't dig any further without putting it into divert again and losing the coverage. Posting in case any of it is useful. Entirely possible it's all one thing you already know about.
(For transparency: I used Claude Opus 5 to help work through the diagnostics and to draft this post. Everything above is from my own box and I've checked the numbers myself.)
