OPNSense momentarily hangs/spikes CPU

[Sam of Ham]

Netflow is still currently disabled - and Suricata is the next highest user 95% of the time.

Beyond that, I see moments of 'hanging' followed by these;
6554   - No Info?
66036 -
PID TT  STAT    TIME COMMAND

Code: [Select]

66036  -  Ss   3:43.93 /usr/local/bin/suricata -D --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml70095 - No Info?

CPU usage looks fine... Load average is a bit high, floating between 0.57 - 1.20, on the first counter. I'm staring at top watching things jump around and rarel seeing usage % spike, same in the dashboard. Again, CPU usage looks fine, which is why this is so stumping!

Wondering if I should revert back to a backup? I'm certainly hoping it's not a hardware issue. it's definitely not temps, those are ~50c if the dashboard is at all accurate (unless it's throttling itself, but I'd assume OPNsense wouldn't run it so low if so - most CPUs would run at max temp until they either slow down to nothing, shut off, or if unlucky, burn up - I'd assume it would be sitting at 80-100, whatever TJM for Celeron J3160s in passive boxex are)

[doktornotor]

Suricata is pretty normal. Don't like the CPU usage - turn it off. Inspecting pretty much every single packet passing through does not come for free really.

[Sam of Ham]

Doesn't explain why it was on just fine and now is hogging CPU. Degradation has to happen, the CPU can't just get out of bed on the wrong side this morning. Still, perhaps it is a matter of 'settling in' to the hardware - unclear.

Anyway naturally off would save resources but I've already shown that it's just as bad off as on, and I can't just keep turning off firewall features until it's working as it should, or I might as well throw the old ISP TP-Link back in and call it a day.

[doktornotor]

Ok, I think it's called mass psychosis. Suricata has had been using CPU intensively since forever but all of a sudden it's hogging now.

[Sam of Ham]

Perhaps I mis-communicated; let me clarify.

What I mean to say is that Suricata is using an average of 1%-2% according to `top`. Thus, unless I'm misreading it entirely (and again, I did mention I'm new, so I could well be! Learning by doing, over here, haha!) I can't see how it is hogging the CPU. I'm only referring to my system here; fully acknowledge that it's a hogging process in general and has been since development.

Additionally, the problem did not go away when disabling Suricata.

Now that doesn't mean it's not Suricata, but it doesn't logically compute to me how it is when it's off and/or only using a tiny portion of CPU.

I'm looking at a CPU graph in GUI that spikes at somewhat-regular intervals, briefly, to 60%/80%/100% and a Discord ping graph (and users listening in on Discord) that either spikes the same way to 200-1000 ping, OR, doesn't even show spikes at all - and yet they're very much audible on the other end, either by complete lost audio, or by massively delayed audio (in some cases, up to 10 seconds late.)

So from my admittedly uneducated standpoint, I see a device that visually looks to be running well within hardware limitations, and yet is seemingly randomly hanging on something. I see Suricata reliably at the top of my `top` list yet only using a few % CPU. I see occasional spikes from other PIDs, but haven't even been able to identify what those PIDs are (as previous - two of them don't even show a line when doing `ps www 12345`) which is why I feel so lost. That's where I'm left right now - with the following questions:

1. What is doing this? What deeper checks can I do to find this out, given `top` isn't necessarily helping me here?
2. Should I revert to a previous saved state? Is it in any way clear that the issue sits with a firmware configuration, or is it more suspect that it's a hardware issue?
3. Gut feelings - should I check over the hardware? Repaste the (arguably probably a bit old!) thermal paste and see if the thing is throttling? Should I simply look for new hardware and call this out of spec (noting that @newsense believed this hardware looked up to spec)?

If it's not also obvious, I've been reading tons of threads elsewhere and not simply waiting for someone else to fix my problem here, but I keep coming up to the same thing - either disabling X service worked, or people using VMs who have allocated more cores or memory and fixed their problems.

I do appreciate that IDS is a hog and could well be the difference-maker here. I just don't want to start disabling things until I have a working system that's been neutered to the point where the old ISP router would have been anyways.

Being a noob operator here, there are a lot of commands and tests that are just not apparent without sinking a week into documentation for all the involved systems, which is where experienced operators who have the commands well learned come in. (Hence the searches returning 'htop' when you simply meant `top` before.) Honest admission of being the "knows enough to be dangerous, not enough to get out of danger" type and I do apologise for the PITA that would be to the more experienced!

[Sam of Ham]

As an update, I've rolled back the system to a config from a few days ago and seem to be having the same issues.

It's such a strange 'ghost' issue - Discord ping graph, Steam downloads, file downloads, even online games all seem to work fine - I'm starting to wonder if it's not CPU, but something interfering with Discord. Still perplexed... Hoping to get to the bottom of this!

if anyone thinks of more console-y stuff I can run to dig into this, it's welcome. Thank you all.

[Sam of Ham]

Interestingly, I see a few reports from pf in the system log - hadn't even noticed these so far as they're hidden in plain sight - on the dashboard, right underneath the network traffic graph I'm actually watching for latency spikes.

I'm... Not sure what these mean. PF appears to be dropping the occasional packet?

Wondering if anyone can interpret these and help edumacate me on what these TCP in/out/state match things mean in this situation.

FYI, find-all-and-replace'd my network's public IP with MY_IP_ADDRESS. LAN IPs and public IPs are fair game.

Code: [Select]

2024-09-06T19:54:12 Notice kernel pf: dropping packet with ip options
2024-09-06T19:53:31 Notice kernel pf: loose state match: TCP in wire: 192.168.1.195:17031 185.90.14.231:443 stack: - [lo=3625498622 high=3625562281 win=1026 modulator=0 wscale=8] [lo=539415783 high=539677228 win=502 modulator=0 wscale=7] 9:9 R seq=539415783 (539415751) ack=3625498622 len=0 ackskew=0 pkts=46:41 dir=out,rev
2024-09-06T19:53:31 Notice kernel pf: loose state match: TCP out wire: 185.90.14.231:443 MY_IP_ADDRESS:35352 stack: 185.90.14.231:443 192.168.1.195:17031 [lo=3625498622 high=3625562281 win=1026 modulator=0 wscale=8] [lo=539415783 high=539677228 win=502 modulator=0 wscale=7] 7:9 R seq=539415783 (539415751) ack=3625498622 len=0 ackskew=0 pkts=45:41 dir=in,rev
2024-09-06T19:53:31 Notice kernel pf: loose state match: TCP in wire: 192.168.1.101:64127 185.90.14.231:443 stack: - [lo=3523303141 high=3523366797 win=516 modulator=0 wscale=8] [lo=2854126245 high=2854257111 win=502 modulator=0 wscale=7] 9:9 R seq=2854126245 (2854126213) ack=3523303141 len=0 ackskew=0 pkts=157:153 dir=out,rev
2024-09-06T19:53:31 Notice kernel pf: loose state match: TCP out wire: 185.90.14.231:443 MY_IP_ADDRESS:51915 stack: 185.90.14.231:443 192.168.1.101:64127 [lo=3523303141 high=3523366797 win=516 modulator=0 wscale=8] [lo=2854126245 high=2854257111 win=502 modulator=0 wscale=7] 9:9 R seq=2854126245 (2854126213) ack=3523303141 len=0 ackskew=0 pkts=156:153 dir=in,rev
2024-09-06T19:52:56 Notice kernel [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=4173380781 (4173380756) ack=2616330636 len=0 ackskew=0 pkts=363:241 dir=out,fwd
2024-09-06T19:52:56 Notice kernel pf: loose state match: pf: loose state match: TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:26704 stack: 9.9.9.9:443 192.168.1.191:44834TCP [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] in wire: 192.168.1.191:44834 9.9.9.9:443 stack: - [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=2616330636 (2616330555) ack=4173380781 len=0 ackskew=0 pkts=364:240 dir=out,rev
2024-09-06T19:52:56 Notice kernel :44834 9.9.9.9:443 stack: - [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=4173380781 (4173380756) ack=2616330636 len=0 ackskew=0 pkts=363:240 dir=in,fwd
2024-09-06T19:52:56 Notice kernel pf: loose state match: pf: loose state match: TCPTCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:26704 stack: 9.9.9.9:443 192.168.1.191:44834 in wire: 192.168.1.191 [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=2616330636 (2616330555) ack=4173380781 len=0 ackskew=0 pkts=363:240 dir=in,rev
2024-09-06T19:52:56 Notice kernel wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=4173380781 (4173380756) ack=2616330636 len=0 ackskew=0 pkts=362:240 dir=out,fwd
2024-09-06T19:52:56 Notice kernel pf: loose state match: pf: loose state match: TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:26704 stack: 9.9.9.9TCP in wire: 192.168.1.191:44834 9.9.9.9:443 stack: - [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7]:443 192.168.1.191:44834 [lo=4173380781 high=4173401492 win=502 modulator=0 [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=2616330636 (2616330555) ack=4173380781 len=0 ackskew=0 pkts=363:239 dir=out,rev
2024-09-06T19:52:56 Notice kernel [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 9:7 R seq=4173380781 (4173380756) ack=2616330636 len=0 ackskew=0 pkts=362:239 dir=in,fwd
2024-09-06T19:52:56 Notice kernel pf: loose state match: TCP inpf: loose state match: wire: 192.168.1.191TCP out:44834 9.9.9.9:443 stack: - wire: 9.9.9.9:443 MY_IP_ADDRESS:26704 stack: 9.9.9.9:443 192.168.1.191:44834 [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 9:7 R seq=2616330636 (2616330555) ack=4173380781 len=0 ackskew=0 pkts=362:239 dir=in,rev
2024-09-06T19:49:58 Notice kernel pf: loose state match: TCP in wire: 192.168.1.191:49740 151.101.193.140:443 stack: - [lo=2462584780 high=2462742988 win=354 modulator=0 wscale=9] [lo=378357056 high=378538279 win=309 modulator=0 wscale=9] 10:10 R seq=378357056 (378334059) ack=2462584780 len=0 ackskew=0 pkts=53:65 dir=out,rev
2024-09-06T19:49:58 Notice kernel pf: loose state match: TCP out wire: 151.101.193.140:443 MY_IP_ADDRESS:58600 stack: 151.101.193.140:443 192.168.1.191:49740 [lo=2462584780 high=2462742988 win=354 modulator=0 wscale=9] [lo=378357056 high=378538279 win=309 modulator=0 wscale=9] 10:10 R seq=378357056 (378334059) ack=2462584780 len=0 ackskew=0 pkts=53:65 dir=in,rev
2024-09-06T19:49:41 Notice kernel pf: loose state match: TCP out wire: 104.21.233.233:443 MY_IP_ADDRESS:30999 stack: 104.21.233.233:443 192.168.1.191:39900 [lo=1430021214 high=1430094942 win=143 modulator=0 wscale=9] [lo=1737975641 high=1738048328 win=9 modulator=0 wscale=13] 9:7 R seq=1430021214 (1430021071) ack=1737975641 len=0 ackskew=0 pkts=8:7 dir=out,fwd
2024-09-06T19:49:41 Notice kernel pf: loose state match: TCP in wire: 192.168.1.191:39900 104.21.233.233:443 stack: - [lo=1430021214 high=1430094799 win=143 modulator=0 wscale=9] [lo=1737975640 high=1738048328 win=9 modulator=0 wscale=13] 9:4 R seq=1430021214 (1430021071) ack=1737975640 len=0 ackskew=0 pkts=8:6 dir=in,fwd
2024-09-06T19:48:55 Notice kernel pf: State failure on: |
2024-09-06T19:48:55 Notice kernel pf: BAD state: TCP out wire: 3.125.149.81:80 MY_IP_ADDRESS:31705 stack: 3.125.149.81:80 192.168.1.163:48580 [lo=3065242015 high=3065296889 win=332 modulator=0 wscale=8] [lo=4024237747 high=4024322676 win=490 modulator=0 wscale=7] 4:4 R seq=4024237747 (4024237684) ack=3065242015 len=0 ackskew=0 pkts=225:158 dir=in,rev
2024-09-06T19:48:52 Notice kernel TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:20913 stack: 9.9.9.9:443 192.168.1.191:46442 [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=1205303738 (1205303713) ack=4183299327 len=0 ackskew=0 pkts=23:17 dir=out,fwd
2024-09-06T19:48:52 Notice kernel pf: loose state match: pf: loose state match: TCP in wire: 192.168.1.191:46442 9.9.9.9:443 stack: - [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=4183299327 (4183299246) ack=1205303738 len=0 ackskew=0 pkts=24:16 dir=out,rev
2024-09-06T19:48:52 Notice kernel wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=1205303738 (1205303713) ack=4183299327 len=0 ackskew=0 pkts=23:16 dir=in,fwd
2024-09-06T19:48:52 Notice kernel pf: loose state match: pf: loose state match: TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:20913 stack: 9.9.9.9:443 192.168.1.191:46442 [lo=1205303738 high=1205324961 win=502 modulator=0TCP in wire: 192.168.1.191:46442 9.9.9.9:443 stack: - [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=4183299327 (4183299246) ack=1205303738 len=0 ackskew=0 pkts=23:16 dir=in,rev
2024-09-06T19:48:52 Notice kernel wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=1205303738 (1205303713) ack=4183299327 len=0 ackskew=0 pkts=22:16 dir=out,fwd
2024-09-06T19:48:52 Notice kernel pf: loose state match: pf: loose state match: TCP in wire: 192.168.1.191:46442 9.9.9.9:443 stack: TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:20913 stack: 9.9.9.9:443 192.168.1.191:46442 [lo=1205303738 high=1205324961 win=502 modulator=0- [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=4183299327 (4183299246) ack=1205303738 len=0 ackskew=0 pkts=23:15 dir=out,rev
2024-09-06T19:48:52 Notice kernel [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 9:7 R seq=4183299327 (4183299246) ack=1205303738 len=0 ackskew=0 pkts=22:15 dir=in,rev
2024-09-06T19:48:52 Notice kernel pf: loose state match: pf: loose state match: TCP out wire: 9.9.9.9TCP in wire: 192.168.1.191:46442 9.9.9.9:443 stack: - [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0:443 MY_IP_ADDRESS:20913 stack: 9.9.9.9:443 192.168.1.191:46442 [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] wscale=8] 9:7 R seq=1205303738 (1205303713) ack=4183299327 len=0 ackskew=0 pkts=22:15 dir=in,fwd
2024-09-06T19:44:04 Notice kernel pf: State failure on: |
2024-09-06T19:44:04 Notice kernel pf: BAD state: TCP out wire: 3.125.149.81:80 MY_IP_ADDRESS:31705 stack: 3.125.149.81:80 192.168.1.163:48580 [lo=3065242015 high=3065296889 win=332 modulator=0 wscale=8] [lo=4024237747 high=4024322676 win=490 modulator=0 wscale=7] 4:4 R seq=4024237747 (4024237684) ack=3065242015 len=0 ackskew=0 pkts=224:158 dir=in,rev
2024-09-06T19:41:12 Notice kernel pf: dropping packet with ip options
2024-09-06T19:39:45 Notice kernel pf: loose state match: TCP in wire: 192.168.1.148:61741 104.74.40.214:443 stack: - [lo=2066156501 high=2066220628 win=1026 modulator=0 wscale=8] [lo=272285939 high=272548595 win=502 modulator=0 wscale=7] 10:10 R seq=272285939 (272284883) ack=2066156501 len=0 ackskew=0 pkts=14:15 dir=out,rev
2024-09-06T19:39:45 Notice kernel pf: loose state match: TCP out wire: 104.74.40.214:443 MY_IP_ADDRESS:64218 stack: 104.74.40.214:443 192.168.1.148:61741 [lo=2066156501 high=2066220628 win=1026 modulator=0 wscale=8] [lo=272285939 high=272548595 win=502 modulator=0 wscale=7] 10:10 R seq=272285939 (272284883) ack=2066156501 len=0 ackskew=0 pkts=14:15 dir=in,rev
2024-09-06T19:34:32 Notice kernel pf: State failure on: |
2024-09-06T19:34:32 Notice kernel pf: BAD state: TCP out wire: 43.245.48.50:443 MY_IP_ADDRESS:12329 stack: 43.245.48.50:443 192.168.1.163:53592 [lo=3962970762 high=3963035247 win=414 modulator=0 wscale=8] [lo=3255328283 high=3255434056 win=510 modulator=0 wscale=7] 4:4 R seq=3255328283 (3255328072) ack=3962970762 len=0 ackskew=0 pkts=44:37 dir=in,rev
2024-09-06T19:33:59 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:59 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:6 dir=in,fwd
2024-09-06T19:33:48 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:48 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=21720 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:5 dir=in,fwd
2024-09-06T19:33:42 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:42 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:5 dir=in,fwd
2024-09-06T19:33:34 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:34 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:4 dir=in,fwd
2024-09-06T19:33:34 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:34 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=21720 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:4 dir=in,fwd
2024-09-06T19:33:30 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:30 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:3 dir=in,fwd
2024-09-06T19:33:30 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:30 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=21720 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:3 dir=in,fwd
2024-09-06T19:33:28 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:28 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:2 dir=in,fwd
2024-09-06T19:33:28 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:28 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=21720 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:2 dir=in,fwd
2024-09-06T19:33:27 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:27 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=85 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:1 dir=in,fwd
2024-09-06T19:33:27 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:27 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=85 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:1 dir=in,fwd
2024-09-06T19:33:27 Notice kernel pf: state reuse TCP out wire: 9.9.9.9:53 MY_IP_ADDRESS:62522 stack: 9.9.9.9:53 192.168.1.192:44770 [lo=664065986 high=664087746 win=502 modulator=0 wscale=7] [lo=665186288 high=665250416 win=85 modulator=0 wscale=8] 9:9 S
2024-09-06T19:33:27 Notice kernel pf: state reuse TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065986 high=664087746 win=502 modulator=0 wscale=7] [lo=665186288 high=665250416 win=85 modulator=0 wscale=8] 9:9 S
2024-09-06T19:33:27 Notice kernel pf: state reuse TCP out wire: 9.9.9.9:53 MY_IP_ADDRESS:41776 stack: 9.9.9.9:53 192.168.1.192:44762 [lo=1914432197 high=1914453957 win=502 modulator=0 wscale=7] [lo=1381881699 high=1381945827 win=85 modulator=0 wscale=8] 9:9 S
2024-09-06T19:33:27 Notice kernel pf: state reuse TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432197 high=1914453957 win=502 modulator=0 wscale=7] [lo=1381881699 high=1381945827 win=85 modulator=0 wscale=8] 9:9 S
2024-09-06T19:33:23 Notice kernel pf: loose state match: TCP out wire: 172.64.150.233:443 MY_IP_ADDRESS:48942 stack: 172.64.150.233:443 192.168.1.191:37534 [lo=3240971823 high=3241045551 win=131 modulator=0 wscale=9] [lo=1631947932 high=1632014491 win=9 modulator=0 wscale=13] 9:7 R seq=3240971823 (3240971680) ack=1631947932 len=0 ackskew=0 pkts=6:5 dir=out,fwd
2024-09-06T19:33:23 Notice kernel pf: loose state match: TCP in wire: 192.168.1.191:37534 172.64.150.233:443 stack: - [lo=3240971823 high=3241045551 win=131 modulator=0 wscale=9] [lo=1631947932 high=1632014491 win=9 modulator=0 wscale=13] 9:7 R seq=3240971823 (3240971680) ack=1631947932 len=0 ackskew=0 pkts=6:5 dir=in,fwd
2024-09-06T19:33:20 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:24016 stack: 52.98.142.136:443 192.168.1.159:57876 [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:20 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57876 52.98.142.136:443 stack: - [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:8114 stack: 52.98.142.136:443 192.168.1.159:57876 [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57876 52.98.142.136:443 stack: - [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:56699 stack: 52.98.142.136:443 192.168.1.159:57875 [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57875 52.98.142.136:443 stack: - [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:49204 stack: 52.98.142.136:443 192.168.1.159:57876 [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57876 52.98.142.136:443 stack: - [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:61140 stack: 52.98.142.136:443 192.168.1.159:57875 [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57875 52.98.142.136:443 stack: - [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:41275 stack: 52.98.142.136:443 192.168.1.159:57876 [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57876 52.98.142.136:443 stack: - [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:38905 stack: 52.98.142.136:443 192.168.1.159:57875 [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57875 52.98.142.136:443 stack: - [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:46832 stack: 52.98.142.136:443 192.168.1.159:57875 [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57875 52.98.142.136:443 stack: - [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:28885 stack: 52.98.140.56:443 192.168.1.159:57874 [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57874 52.98.140.56:443 stack: - [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:27570 stack: 52.98.140.56:443 192.168.1.159:57874 [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57874 52.98.140.56:443 stack: - [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:11784 stack: 52.98.140.56:443 192.168.1.159:57873 [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57873 52.98.140.56:443 stack: - [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:57899 stack: 52.98.140.56:443 192.168.1.159:57874 [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57874 52.98.140.56:443 stack: - [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:25304 stack: 52.98.140.56:443 192.168.1.159:57873 [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57873 52.98.140.56:443 stack: - [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:1100 stack: 52.98.140.56:443 192.168.1.159:57874 [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57874 52.98.140.56:443 stack: - [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:16579 stack: 52.98.140.56:443 192.168.1.159:57873 [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57873 52.98.140.56:443 stack: - [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:12072 stack: 52.98.140.56:443 192.168.1.159:57873 [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57873 52.98.140.56:443 stack: - [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:59506 stack: 40.99.134.24:443 192.168.1.159:57872 [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57872 40.99.134.24:443 stack: - [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:46007 stack: 40.99.134.24:443 192.168.1.159:57872 [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57872 40.99.134.24:443 stack: - [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:37695 stack: 40.99.134.24:443 192.168.1.159:57871 [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57871 40.99.134.24:443 stack: - [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:55784 stack: 40.99.134.24:443 192.168.1.159:57872 [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57872 40.99.134.24:443 stack: - [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:12364 stack: 40.99.134.24:443 192.168.1.159:57871 [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57871 40.99.134.24:443 stack: - [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:45599 stack: 40.99.134.24:443 192.168.1.159:57872 [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57872 40.99.134.24:443 stack: - [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:31168 stack: 40.99.134.24:443 192.168.1.159:57871 [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57871 40.99.134.24:443 stack: - [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:28098 stack: 40.99.134.24:443 192.168.1.159:57871 [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57871 40.99.134.24:443 stack: - [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:18378 stack: 52.98.140.24:443 192.168.1.159:57870 [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57870 52.98.140.24:443 stack: - [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: loose state match: TCP in wire: 192.168.1.159:57841 52.113.194.132:443 stack: - [lo=508273165 high=512467981 win=258 modulator=0 wscale=8] [lo=2514990012 high=2515056060 win=16386 modulator=0 wscale=8] 10:10 R seq=2514990012 (2514987340) ack=508273165 len=0 ackskew=0 pkts=35:71 dir=out,rev
2024-09-06T19:33:13 Notice kernel pf: loose state match: TCP out wire: 52.113.194.132:443 MY_IP_ADDRESS:42597 stack: 52.113.194.132:443 192.168.1.159:57841 [lo=508273165 high=512467981 win=258 modulator=0 wscale=8] [lo=2514990012 high=2515056060 win=16386 modulator=0 wscale=8] 10:10 R seq=2514990012 (2514987340) ack=508273165 len=0 ackskew=0 pkts=35:71 dir=in,rev
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:9686 stack: 52.98.140.24:443 192.168.1.159:57870 [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57870 52.98.140.24:443 stack: - [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:20532 stack: 52.98.140.24:443 192.168.1.159:57869 [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57869 52.98.140.24:443 stack: - [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:30982 stack: 52.98.140.24:443 192.168.1.159:57870 [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57870 52.98.140.24:443 stack: - [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:34651 stack: 52.98.140.24:443 192.168.1.159:57869 [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57869 52.98.140.24:443 stack: - [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:11 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:10533 stack: 52.98.140.24:443 192.168.1.159:57870 [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:11 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57870 52.98.140.24:443 stack: - [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:11 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:4752 stack: 52.98.140.24:443 192.168.1.159:57869 [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:11 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57869 52.98.140.24:443 stack: - [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:65217 stack: 52.98.140.24:443 192.168.1.159:57869 [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57869 52.98.140.24:443 stack: - [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:24747 stack: 52.98.143.136:443 192.168.1.159:57865 [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57865 52.98.143.136:443 stack: - [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:33258 stack: 52.98.143.136:443 192.168.1.159:57865 [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57865 52.98.143.136:443 stack: - [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:6125 stack: 52.98.143.136:443 192.168.1.159:57864 [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57864 52.98.143.136:443 stack: - [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:23050 stack: 52.98.143.136:443 192.168.1.159:57863 [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57863 52.98.143.136:443 stack: - [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:5126 stack: 52.98.143.136:443 192.168.1.159:57865 [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57865 52.98.143.136:443 stack: - [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:14239 stack: 52.98.143.136:443 192.168.1.159:57864 [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57864 52.98.143.136:443 stack: - [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:63286 stack: 52.98.143.136:443 192.168.1.159:57863 [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57863 52.98.143.136:443 stack: - [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:36042 stack: 52.98.143.136:443 192.168.1.159:57865 [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57865 52.98.143.136:443 stack: - [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:25502 stack: 52.98.143.136:443 192.168.1.159:57864 [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57864 52.98.143.136:443 stack: - [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:08 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:30954 stack: 52.98.143.136:443 192.168.1.159:57863 [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:08 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57863 52.98.143.136:443 stack: - [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:08 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:13292 stack: 52.98.143.136:443 192.168.1.159:57862 [lo=3292658861 high=3292667074 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:08 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57862 52.98.143.136:443 stack: - [lo=3292658861 high=3292667074 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S


[meyergru]

Maybe the excessive logging is causing the spikes. Also, I wonder if the dreaded FreeBSD SA causes this.

Could you try

Code: [Select]

opnsense-update -zkr 24.7.3-no_sa

and reboot?

[Sam of Ham]

Big excuse mes, but what's the FreeBSD SA?

I do notice I somehow turned on the additional logging at some point and cannot for the life of me work that out. It seems counterintuitive - logging (thing to find problems) causing problems (helped by logging)... Ha. Makes sense though - every action doubled.

Definitely can run that - can you explain what it does?

Thanks a bunch!

[Sam of Ham]

Oh also, worth noting I'm not on 24.7 as yet, still 24.1 - I saw the big thread about the IPv6/ICMP stuff (very fascinating, fwiw) and thought I'd avoid for a few weeks until some launch bugs were worked out. Is that what you mean? Would it matter on 24.1 Sav Shark?

[meyergru]

No, I thought you were current.

[Sam of Ham]

I could do an upgrade if you think it might help out here?

[meyergru]

No real need there, I just thought that maybe excessive logging of newly-introduced pf: messages created your CPU spikes. Im may still be the case, however, that kernel only eliminates a certain part of those messages. As I do not have logging for debug messages, I cannot say if those messages have increased and if so, when.

You could try to lower your logging to Debug: ...only for serious errors under Firewall: Settings: Advanced to try if you did not already have that.

[Sam of Ham]

Thank you @meyergru for your help.

I'm not sure if it fully helps, but I tried stopping the logging daemon entirely for a while - no change. I also stopped Suricata again but there really isn't anything else I can think of to stop that wouldn't compromise the device's functionality or security.

As an update, here's where things are up to. Over the weekend I took a clone of the disk (just for solid backup certainty - thinking I try the major upgrade and see if that somehow fixes this) and repasted the CPU. It wasn't showing extreme thermals, but not being familiar with the J3160, I wasn't sure if it perhaps had a TJM of 50/60 degrees instead of 90/100. It was preeeety crusty and dry, so that wasn't a bad shout. Unfortunately, no change.

My network has an NTD which terminates coax to Ethernet for the WAN, so I took the chance to reboot everything else too, including the switch, since sometimes some switches can get overloaded (and while it's not likely, the fact that I have similar dropping issues on both LAN [over EoP injectors] and WiFi could indiate the switch, since both go into that.)

I did try the old ISP router over dinner but I think I wired it wrong because it never came up, but I intend to test to see if my connection is solid on it as well. That would really narrow things down to either OPNsense or it's hardware. Beyond that... MAYBE a fresh install would prove it's this current firmware/config?

I'm starting to think that perhaps I'm just at the limit of this old thing's capabilities, for whatever reason. Perhaps single-core spikes throttle it or perhaps I have a bad binned chip... I can't tell. I've combed through OPNsense's settings a few times to see where something could have gone wrong, and short of rolling back another config to something, say, over a month old, I'm out of ideas.

Thanks again as always to everyone for helping me out here. Let me know if you have any last ideas!

[Sam of Ham]

I've upgraded to the latest version of OPNSense (24.7.3_1) successfully (side-note; props for the smooth upgrade process, OPNsense team!!) and will test tonight over the usual workloads. It's great to see more granular visuals and get a better at-a-glance understanding of system interrupts and temperatures here!

I will also test with the ISP router when I get the chance one of these nights - just snowed in with work OOO. If THAT fails, then I'm looking to grab one of the following devices as a replacement with the goal to upgrade to 2-4 Intel NICs and a display-supporting USB-C port (for easy portable screen console viewing instead of having to lug a whole display with power and HDMI up the freaking wall).

I can't dump enough money into this to go up to the ~$600-$800 that a decent Protectli device would do me but am trying to get as close as possible to that level.

HUNSN RH02
HUNSN JR03
HUNSN RJ12
TRIGKEY Green G5
Protectli Vault FW2B
Protectli Vault Pro VP2420