Hi all, and thanks in advance for your help.
Following on from recent issues with ZenArmor (Python 3.9 issues, see https://forum.opnsense.org/index.php?topic=42074.0;topicseen and https://forum.opnsense.org/index.php?topic=41932.0;topicseen) which I have now removed until it's proven stable again, I keep having momentary 'hangs' in my OPNsense system that result in, effectively, significant packet loss and lag.
My system's health checks come up OK, and it's a toss-up as to whether something like Discord's ping graph or an ping google.com -t shows any dropouts... But the effects are very much there in anything constant, like a Discord call or an online game.
I AM running OPNSense on an older box, a Qotom Mini PC with a J3160 and 8 GB DDR3 in it, so that could be a factor, but after initially rolling out OPNsense, I was having a great time... I can't work out why it's started to go bad, unless I just happened to get a bad unit or something. (It WAS second hand.)
I'm not the best with Linux or FreeBSD so this is an open call for help... What can I do do diagnose this, and learn as I go, so that I can either repair whatever's unhappy or at least confirm that I need to buy something new?
Thank you all!
You can confirm with top in the console/ssh which process hangs.
If you don't have large lists processing either as Aliases or in Unbound, next culprit would be the reporting DB (and maybe the DNS one.)
If not using the reporting then disable it altogether in Reporting - Settings, else at the very least reset RRD and Netflow data.
You have a fairly decent box, things should run quite smoothly once the issue pegging the CPU is addressed.
Hey! Thanks for your reply!
When you say top - do you mean pfTop via the console? I've SSH'd into it and have been looking at that/firewall log but can't see anything immediately problematic. pfTop is a lot of netstat-style connections so I don't think it's here.
I've reset NetFlow (I was listening on all interfaces [LAN, WAN, TAILSCALE, UNTRUSTED] and changed that to just LAN/WAN. Doesn't look like much change yet, still getting random timeouts in constant ping to google.
I have a Pihole that's my DNS and DHCP server - could there still be some logging issues on the OPNsense box anyways?
The DDR3 Celeron thing seemed to be fine when I got it and it did indeed work alright for a bit, but now it's unhappy, so I wasn't sure if I was just getting good performance for as long as it was 'brand new' so to speak. Glad to hear it seems to be up to spec!
top, not pftop.
I'm going to go out on a limb here and assume you mean htop?
This is the top result when searching for it https://forum.opnsense.org/index.php?topic=15011.0 and it's from 2019, so perhaps out of date.
Franco mentions using $ pkg install -A autoconf automake libtool and the most recent reply says to [sic] add the mimugmail repo and htop via #pkg install htop but neither commands work for me.
Sorry, I know it's frustrating having to spell it out for a noob, but if you could spare a sec to educate me I'd be super grateful!
Tell a lie, I realised (or more accurately, wondered) that the "$ " was unnecessary. Seems to work better now but doesn't contain htop. I get this:
root@OPNsense:~ # pkg install htop
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'htop' have been found in the repositories
root@OPNsense:~ #
Same for top.
Type top into shell. You do not need htop. You do not need to compile anything.
Oh... Cool, great to know!
Here's a snippet of the output. Nothing immediately jumps out at me as problematic?
last pid: 32400; load averages: 0.48, 0.54, 0.50 up 2+05:37:59 19:59:55
46 processes: 1 running, 45 sleeping
CPU: 0.4% user, 0.0% nice, 0.2% system, 0.1% interrupt, 99.3% idle
Mem: 61M Active, 2255M Inact, 2195M Laundry, 3146M Wired, 581M Buf, 318M Free
Swap: 8192M Total, 579M Used, 7613M Free, 7% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
56942 root 7 20 0 6486M 1840M nanslp 2 53:22 1.57% suricata
73162 root 1 20 0 14M 3288K CPU3 3 0:00 0.09% top
89890 root 1 20 0 27M 13M select 1 0:53 0.01% python3.11
88541 root 1 20 0 26M 12M select 2 0:30 0.01% python3.11
17965 root 1 33 0 13M 1912K wait 1 0:43 0.01% sh
56359 root 1 20 0 19M 9672K select 3 0:00 0.01% sshd-session
8833 root 1 20 0 23M 5972K select 3 0:17 0.01% ntpd
40019 root 1 21 0 52M 35M nanslp 3 110:00 0.01% python3.11
28497 root 1 20 0 23M 3444K select 3 0:04 0.01% mpd5
53741 root 1 20 0 22M 8128K kqread 1 0:35 0.00% lighttpd
84435 root 1 20 0 13M 1908K bpf 1 0:50 0.00% filterlog
35995 root 2 20 0 20M 3496K nanslp 3 0:05 0.00% monit
270 root 1 20 0 86M 31M accept 0 10:29 0.00% python3.11
10839 root 3 20 0 54M 12M kqread 2 3:32 0.00% syslog-ng
8032 root 12 20 0 1262M 31M uwait 1 2:27 0.00% tailscaled
45415 _flowd 1 20 0 12M 1632K select 1 0:07 0.00% flowd
266 root 1 52 0 26M 12M wait 3 0:03 0.00% python3.11
16035 root 1 32 0 13M 1864K nanslp 1 0:02 0.00% cron
2869 root 1 24 0 63M 31M accept 1 0:02 0.00% php-cgi
82580 root 1 21 0 63M 31M accept 1 0:02 0.00% php-cgi
15862 root 1 20 0 63M 30M accept 1 0:01 0.00% php-cgi
20014 nobody 1 20 0 12M 1788K sbwait 1 0:01 0.00% samplicate
47672 root 1 20 0 63M 30M accept 3 0:01 0.00% php-cgi
17033 root 1 20 0 12M 1276K piperd 0 0:01 0.00% daemon
54518 root 1 20 0 59M 25M wait 0 0:00 0.00% php-cgi
95682 root 1 20 0 63M 30M accept 3 0:00 0.00% php-cgi
54155 root 1 20 0 59M 25M wait 0 0:00 0.00% php-cgi
20149 root 1 39 0 63M 30M accept 3 0:00 0.00% php-cgi
7768 root 1 20 0 12M 1272K piperd 1 0:00 0.00% daemon
589 root 1 20 0 11M 824K select 2 0:00 0.00% devd
48252 root 1 25 0 19M 9376K select 3 0:00 0.00% sshd-session
16447 root 1 20 0 19M 8144K select 1 0:00 0.00% sshd
71192 root 1 37 0 13M 3532K pause 2 0:00 0.00% csh
56558 root 1 45 0 13M 2612K wait 2 0:00 0.00% sh
58114 root 1 52 0 12M 1544K ttyin 2 0:00 0.00% getty
57765 root 1 52 0 12M 1548K ttyin 3 0:00 0.00% getty
60472 root 1 52 0 12M 1544K ttyin 2 0:00 0.00% getty
58982 root 1 52 0 12M 1548K ttyin 3 0:00 0.00% getty
61297 root 1 52 0 12M 1544K ttyin 3 0:00 0.00% getty
59071 root 1 52 0 12M 1544K ttyin 2 0:00 0.00% getty
59274 root 1 52 0 12M 1548K ttyin 2 0:00 0.00% getty
60338 root 1 52 0 12M 1544K ttyin 2 0:00 0.00% getty
45172 root 1 20 0 12M 1484K sbwait 2 0:00 0.00% flowd
10492 root 1 52 0 23M 7896K wait 2 0:00 0.00% syslog-ng
32400 root 1 33 0 12M 1800K nanslp 2 0:00 0.00% sleep
19737 root 1 52 0 12M 1872K piperd 1 0:00 0.00% daemon
Every now and then, it freezes for a sec, and something using Python 3.11 jumps to the top:
last pid: 46555; load averages: 0.37, 0.40, 0.43 up 2+05:45:04 20:07:00
49 processes: 3 running, 46 sleeping
CPU: 16.8% user, 0.0% nice, 2.5% system, 0.1% interrupt, 80.7% idle
Mem: 84M Active, 2256M Inact, 2195M Laundry, 3147M Wired, 581M Buf, 299M Free
Swap: 8192M Total, 579M Used, 7613M Free, 7% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
40019 root 1 24 0 52M 35M CPU3 3 110:11 19.54% python3.11
56942 root 7 20 0 6486M 1840M nanslp 2 53:29 1.53% suricata
10580 root 1 20 0 14M 3292K CPU2 2 0:00 0.08% top
17965 root 1 52 0 13M 1912K wait 3 0:43 0.04% sh
16035 root 1 20 0 13M 1864K nanslp 1 0:02 0.02% cron
56359 root 1 20 0 19M 9672K select 1 0:00 0.02% sshd-session
88541 root 1 20 0 26M 12M select 3 0:30 0.02% python3.11
10839 root 3 20 0 54M 12M kqread 3 3:32 0.02% syslog-ng
89890 root 1 20 0 27M 13M select 2 0:53 0.02% python3.11
8833 root 1 20 0 23M 5972K select 1 0:17 0.01% ntpd
53741 root 1 20 0 22M 8128K kqread 2 0:35 0.00% lighttpd
35995 root 2 20 0 20M 3496K nanslp 0 0:05 0.00% monit
84435 root 1 20 0 13M 1908K bpf 2 0:50 0.00% filterlog
270 root 1 20 0 86M 31M accept 2 10:29 0.00% python3.11
8032 root 12 20 0 1262M 31M uwait 2 2:27 0.00% tailscaled
45415 _flowd 1 20 0 12M 1632K select 1 0:07 0.00% flowd
28497 root 1 20 0 23M 3444K select 1 0:04 0.00% mpd5
266 root 1 52 0 26M 12M wait 3 0:03 0.00% python3.11
2869 root 1 24 0 63M 31M accept 1 0:02 0.00% php-cgi
82580 root 1 21 0 63M 31M accept 1 0:02 0.00% php-cgi
15862 root 1 20 0 63M 30M accept 1 0:01 0.00% php-cgi
20014 nobody 1 20 0 12M 1788K sbwait 1 0:01 0.00% samplicate
47672 root 1 20 0 63M 30M accept 3 0:01 0.00% php-cgi
45633 root 1 74 0 58M 39M CPU0 0 0:01 0.00% python3.11
17033 root 1 20 0 12M 1276K piperd 3 0:01 0.00% daemon
54518 root 1 20 0 59M 25M wait 0 0:00 0.00% php-cgi
95682 root 1 20 0 63M 30M accept 3 0:00 0.00% php-cgi
54155 root 1 20 0 59M 25M wait 0 0:00 0.00% php-cgi
20149 root 1 39 0 63M 30M accept 3 0:00 0.00% php-cgi
7768 root 1 20 0 12M 1272K piperd 1 0:00 0.00% daemon
589 root 1 20 0 11M 824K select 0 0:00 0.00% devd
48252 root 1 25 0 19M 9376K select 3 0:00 0.00% sshd-session
16447 root 1 20 0 19M 8144K select 1 0:00 0.00% sshd
6322 root 1 20 0 13M 3536K pause 1 0:00 0.00% csh
56558 root 1 20 0 13M 2612K wait 3 0:00 0.00% sh
45497 root 1 22 0 13M 2208K wait 1 0:00 0.00% flock
45127 root 1 21 0 13M 1960K piperd 3 0:00 0.00% cron
58114 root 1 52 0 12M 1544K ttyin 2 0:00 0.00% getty
57765 root 1 52 0 12M 1548K ttyin 3 0:00 0.00% getty
60472 root 1 52 0 12M 1544K ttyin 2 0:00 0.00% getty
58982 root 1 52 0 12M 1548K ttyin 3 0:00 0.00% getty
61297 root 1 52 0 12M 1544K ttyin 3 0:00 0.00% getty
59071 root 1 52 0 12M 1544K ttyin 2 0:00 0.00% getty
59274 root 1 52 0 12M 1548K ttyin 2 0:00 0.00% getty
60338 root 1 52 0 12M 1544K ttyin 2 0:00 0.00% getty
45172 root 1 20 0 12M 1484K sbwait 2 0:00 0.00% flowd
10492 root 1 52 0 23M 7896K wait 2 0:00 0.00% syslog-ng
45699 root 1 52 0 12M 1796K nanslp 2 0:00 0.00% sleep
19737 root 1 52 0 12M 1872K piperd 1 0:00 0.00% daemon
Though what that means, I'm not super sure. Could the whole ZenArmor Python debacle have left some problematic Python in place or something?
You can see the commandline via "ps www 40019" or whatever PID is hogging your CPU.
Okay, good call. 40019 appears to be netflow. What does "Time" mean here? I'd assume time running, but the 'time' for that PID isn't changing in seconds. Is it time hung?
PID TT STAT TIME COMMAND
40019 - Rs 114:04.44 /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.11)
I've reset NetFlow data and turned off the interfaces (leaving both saying "nothing selected") hoping that that will free things up a bit. Can you guys recommend anything more? Again, apologies for the newbieism here - trying to learn as best I can as I go, both from you guys and lots of searching!
40019 now no longer shows up in my `top` list but I still get varied pings to google and fairly regular spikes of 200-600 in Discord. I'm lost! I don't know what changed between good performance time and now, except for ZenArmor's stuff!
newsense already told you to at least repair, better reset the netflow database. With high CPU load after an update, this is always the first thing to try.
Okay, well, I did reset the Netflow database. I'll try the repair and see if that helps. Forgive me but I would have assumed resetting the database and/or just disabling the thing could have shown at least a temporary change, thus indicating that this service is actually the problem. All good - will see if the repair works, and report back! Thanks!
So far after a reset and repair of the database and a reboot (later on - did that today) seems to be no major change. My ping graph in Discord looks like the Frasier titlecard. I'm stumped! Any further ideas?
Again, and as always, big thanks to everyone.
It's all the same exercise again. Find out what's spiking your CPU. And that netflow feature should be disabled (untick any interfaces enabled there, reset the db, maybe even reboot after that. (No idea what's the cool factor behind these services without an Enable/Disable checkbox. NTPD comes to mind as well.)
Netflow is still currently disabled - and Suricata is the next highest user 95% of the time.
Beyond that, I see moments of 'hanging' followed by these;
6554 - No Info?
66036 -
PID TT STAT TIME COMMAND
66036 - Ss 3:43.93 /usr/local/bin/suricata -D --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml
70095 - No Info?
CPU usage looks fine... Load average is a bit high, floating between 0.57 - 1.20, on the first counter. I'm staring at top watching things jump around and rarel seeing usage % spike, same in the dashboard. Again, CPU usage looks fine, which is why this is so stumping!
Wondering if I should revert back to a backup? I'm certainly hoping it's not a hardware issue. it's definitely not temps, those are ~50c if the dashboard is at all accurate (unless it's throttling itself, but I'd assume OPNsense wouldn't run it so low if so - most CPUs would run at max temp until they either slow down to nothing, shut off, or if unlucky, burn up - I'd assume it would be sitting at 80-100, whatever TJM for Celeron J3160s in passive boxex are)
Suricata is pretty normal. Don't like the CPU usage - turn it off. Inspecting pretty much every single packet passing through does not come for free really.
Doesn't explain why it was on just fine and now is hogging CPU. Degradation has to happen, the CPU can't just get out of bed on the wrong side this morning. Still, perhaps it is a matter of 'settling in' to the hardware - unclear.
Anyway naturally off would save resources but I've already shown that it's just as bad off as on, and I can't just keep turning off firewall features until it's working as it should, or I might as well throw the old ISP TP-Link back in and call it a day.
Ok, I think it's called mass psychosis. Suricata has had been using CPU intensively since forever but all of a sudden it's hogging now.
Perhaps I mis-communicated; let me clarify.
What I mean to say is that Suricata is using an average of 1%-2% according to `top`. Thus, unless I'm misreading it entirely (and again, I did mention I'm new, so I could well be! Learning by doing, over here, haha!) I can't see how it is hogging the CPU. I'm only referring to my system here; fully acknowledge that it's a hogging process in general and has been since development.
Additionally, the problem did not go away when disabling Suricata.
Now that doesn't mean it's not Suricata, but it doesn't logically compute to me how it is when it's off and/or only using a tiny portion of CPU.
I'm looking at a CPU graph in GUI that spikes at somewhat-regular intervals, briefly, to 60%/80%/100% and a Discord ping graph (and users listening in on Discord) that either spikes the same way to 200-1000 ping, OR, doesn't even show spikes at all - and yet they're very much audible on the other end, either by complete lost audio, or by massively delayed audio (in some cases, up to 10 seconds late.)
So from my admittedly uneducated standpoint, I see a device that visually looks to be running well within hardware limitations, and yet is seemingly randomly hanging on something. I see Suricata reliably at the top of my `top` list yet only using a few % CPU. I see occasional spikes from other PIDs, but haven't even been able to identify what those PIDs are (as previous - two of them don't even show a line when doing `ps www 12345`) which is why I feel so lost. That's where I'm left right now - with the following questions:
1. What is doing this? What deeper checks can I do to find this out, given `top` isn't necessarily helping me here?
2. Should I revert to a previous saved state? Is it in any way clear that the issue sits with a firmware configuration, or is it more suspect that it's a hardware issue?
3. Gut feelings - should I check over the hardware? Repaste the (arguably probably a bit old!) thermal paste and see if the thing is throttling? Should I simply look for new hardware and call this out of spec (noting that @newsense believed this hardware looked up to spec)?
If it's not also obvious, I've been reading tons of threads elsewhere and not simply waiting for someone else to fix my problem here, but I keep coming up to the same thing - either disabling X service worked, or people using VMs who have allocated more cores or memory and fixed their problems.
I do appreciate that IDS is a hog and could well be the difference-maker here. I just don't want to start disabling things until I have a working system that's been neutered to the point where the old ISP router would have been anyways.
Being a noob operator here, there are a lot of commands and tests that are just not apparent without sinking a week into documentation for all the involved systems, which is where experienced operators who have the commands well learned come in. (Hence the searches returning 'htop' when you simply meant `top` before.) Honest admission of being the "knows enough to be dangerous, not enough to get out of danger" type and I do apologise for the PITA that would be to the more experienced!
As an update, I've rolled back the system to a config from a few days ago and seem to be having the same issues.
It's such a strange 'ghost' issue - Discord ping graph, Steam downloads, file downloads, even online games all seem to work fine - I'm starting to wonder if it's not CPU, but something interfering with Discord. Still perplexed... Hoping to get to the bottom of this!
if anyone thinks of more console-y stuff I can run to dig into this, it's welcome. Thank you all.
Interestingly, I see a few reports from pf in the system log - hadn't even noticed these so far as they're hidden in plain sight - on the dashboard, right underneath the network traffic graph I'm actually watching for latency spikes.
I'm... Not sure what these mean. PF appears to be dropping the occasional packet?
Wondering if anyone can interpret these and help edumacate me on what these TCP in/out/state match things mean in this situation.
FYI, find-all-and-replace'd my network's public IP with MY_IP_ADDRESS. LAN IPs and public IPs are fair game.
2024-09-06T19:54:12 Notice kernel pf: dropping packet with ip options
2024-09-06T19:53:31 Notice kernel pf: loose state match: TCP in wire: 192.168.1.195:17031 185.90.14.231:443 stack: - [lo=3625498622 high=3625562281 win=1026 modulator=0 wscale=8] [lo=539415783 high=539677228 win=502 modulator=0 wscale=7] 9:9 R seq=539415783 (539415751) ack=3625498622 len=0 ackskew=0 pkts=46:41 dir=out,rev
2024-09-06T19:53:31 Notice kernel pf: loose state match: TCP out wire: 185.90.14.231:443 MY_IP_ADDRESS:35352 stack: 185.90.14.231:443 192.168.1.195:17031 [lo=3625498622 high=3625562281 win=1026 modulator=0 wscale=8] [lo=539415783 high=539677228 win=502 modulator=0 wscale=7] 7:9 R seq=539415783 (539415751) ack=3625498622 len=0 ackskew=0 pkts=45:41 dir=in,rev
2024-09-06T19:53:31 Notice kernel pf: loose state match: TCP in wire: 192.168.1.101:64127 185.90.14.231:443 stack: - [lo=3523303141 high=3523366797 win=516 modulator=0 wscale=8] [lo=2854126245 high=2854257111 win=502 modulator=0 wscale=7] 9:9 R seq=2854126245 (2854126213) ack=3523303141 len=0 ackskew=0 pkts=157:153 dir=out,rev
2024-09-06T19:53:31 Notice kernel pf: loose state match: TCP out wire: 185.90.14.231:443 MY_IP_ADDRESS:51915 stack: 185.90.14.231:443 192.168.1.101:64127 [lo=3523303141 high=3523366797 win=516 modulator=0 wscale=8] [lo=2854126245 high=2854257111 win=502 modulator=0 wscale=7] 9:9 R seq=2854126245 (2854126213) ack=3523303141 len=0 ackskew=0 pkts=156:153 dir=in,rev
2024-09-06T19:52:56 Notice kernel [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=4173380781 (4173380756) ack=2616330636 len=0 ackskew=0 pkts=363:241 dir=out,fwd
2024-09-06T19:52:56 Notice kernel pf: loose state match: pf: loose state match: TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:26704 stack: 9.9.9.9:443 192.168.1.191:44834TCP [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] in wire: 192.168.1.191:44834 9.9.9.9:443 stack: - [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=2616330636 (2616330555) ack=4173380781 len=0 ackskew=0 pkts=364:240 dir=out,rev
2024-09-06T19:52:56 Notice kernel :44834 9.9.9.9:443 stack: - [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=4173380781 (4173380756) ack=2616330636 len=0 ackskew=0 pkts=363:240 dir=in,fwd
2024-09-06T19:52:56 Notice kernel pf: loose state match: pf: loose state match: TCPTCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:26704 stack: 9.9.9.9:443 192.168.1.191:44834 in wire: 192.168.1.191 [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=2616330636 (2616330555) ack=4173380781 len=0 ackskew=0 pkts=363:240 dir=in,rev
2024-09-06T19:52:56 Notice kernel wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=4173380781 (4173380756) ack=2616330636 len=0 ackskew=0 pkts=362:240 dir=out,fwd
2024-09-06T19:52:56 Notice kernel pf: loose state match: pf: loose state match: TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:26704 stack: 9.9.9.9TCP in wire: 192.168.1.191:44834 9.9.9.9:443 stack: - [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7]:443 192.168.1.191:44834 [lo=4173380781 high=4173401492 win=502 modulator=0 [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 10:10 R seq=2616330636 (2616330555) ack=4173380781 len=0 ackskew=0 pkts=363:239 dir=out,rev
2024-09-06T19:52:56 Notice kernel [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 9:7 R seq=4173380781 (4173380756) ack=2616330636 len=0 ackskew=0 pkts=362:239 dir=in,fwd
2024-09-06T19:52:56 Notice kernel pf: loose state match: TCP inpf: loose state match: wire: 192.168.1.191TCP out:44834 9.9.9.9:443 stack: - wire: 9.9.9.9:443 MY_IP_ADDRESS:26704 stack: 9.9.9.9:443 192.168.1.191:44834 [lo=4173380781 high=4173401492 win=502 modulator=0 wscale=7] [lo=2616330636 high=2616394683 win=85 modulator=0 wscale=8] 9:7 R seq=2616330636 (2616330555) ack=4173380781 len=0 ackskew=0 pkts=362:239 dir=in,rev
2024-09-06T19:49:58 Notice kernel pf: loose state match: TCP in wire: 192.168.1.191:49740 151.101.193.140:443 stack: - [lo=2462584780 high=2462742988 win=354 modulator=0 wscale=9] [lo=378357056 high=378538279 win=309 modulator=0 wscale=9] 10:10 R seq=378357056 (378334059) ack=2462584780 len=0 ackskew=0 pkts=53:65 dir=out,rev
2024-09-06T19:49:58 Notice kernel pf: loose state match: TCP out wire: 151.101.193.140:443 MY_IP_ADDRESS:58600 stack: 151.101.193.140:443 192.168.1.191:49740 [lo=2462584780 high=2462742988 win=354 modulator=0 wscale=9] [lo=378357056 high=378538279 win=309 modulator=0 wscale=9] 10:10 R seq=378357056 (378334059) ack=2462584780 len=0 ackskew=0 pkts=53:65 dir=in,rev
2024-09-06T19:49:41 Notice kernel pf: loose state match: TCP out wire: 104.21.233.233:443 MY_IP_ADDRESS:30999 stack: 104.21.233.233:443 192.168.1.191:39900 [lo=1430021214 high=1430094942 win=143 modulator=0 wscale=9] [lo=1737975641 high=1738048328 win=9 modulator=0 wscale=13] 9:7 R seq=1430021214 (1430021071) ack=1737975641 len=0 ackskew=0 pkts=8:7 dir=out,fwd
2024-09-06T19:49:41 Notice kernel pf: loose state match: TCP in wire: 192.168.1.191:39900 104.21.233.233:443 stack: - [lo=1430021214 high=1430094799 win=143 modulator=0 wscale=9] [lo=1737975640 high=1738048328 win=9 modulator=0 wscale=13] 9:4 R seq=1430021214 (1430021071) ack=1737975640 len=0 ackskew=0 pkts=8:6 dir=in,fwd
2024-09-06T19:48:55 Notice kernel pf: State failure on: |
2024-09-06T19:48:55 Notice kernel pf: BAD state: TCP out wire: 3.125.149.81:80 MY_IP_ADDRESS:31705 stack: 3.125.149.81:80 192.168.1.163:48580 [lo=3065242015 high=3065296889 win=332 modulator=0 wscale=8] [lo=4024237747 high=4024322676 win=490 modulator=0 wscale=7] 4:4 R seq=4024237747 (4024237684) ack=3065242015 len=0 ackskew=0 pkts=225:158 dir=in,rev
2024-09-06T19:48:52 Notice kernel TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:20913 stack: 9.9.9.9:443 192.168.1.191:46442 [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=1205303738 (1205303713) ack=4183299327 len=0 ackskew=0 pkts=23:17 dir=out,fwd
2024-09-06T19:48:52 Notice kernel pf: loose state match: pf: loose state match: TCP in wire: 192.168.1.191:46442 9.9.9.9:443 stack: - [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=4183299327 (4183299246) ack=1205303738 len=0 ackskew=0 pkts=24:16 dir=out,rev
2024-09-06T19:48:52 Notice kernel wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=1205303738 (1205303713) ack=4183299327 len=0 ackskew=0 pkts=23:16 dir=in,fwd
2024-09-06T19:48:52 Notice kernel pf: loose state match: pf: loose state match: TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:20913 stack: 9.9.9.9:443 192.168.1.191:46442 [lo=1205303738 high=1205324961 win=502 modulator=0TCP in wire: 192.168.1.191:46442 9.9.9.9:443 stack: - [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=4183299327 (4183299246) ack=1205303738 len=0 ackskew=0 pkts=23:16 dir=in,rev
2024-09-06T19:48:52 Notice kernel wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=1205303738 (1205303713) ack=4183299327 len=0 ackskew=0 pkts=22:16 dir=out,fwd
2024-09-06T19:48:52 Notice kernel pf: loose state match: pf: loose state match: TCP in wire: 192.168.1.191:46442 9.9.9.9:443 stack: TCP out wire: 9.9.9.9:443 MY_IP_ADDRESS:20913 stack: 9.9.9.9:443 192.168.1.191:46442 [lo=1205303738 high=1205324961 win=502 modulator=0- [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 10:10 R seq=4183299327 (4183299246) ack=1205303738 len=0 ackskew=0 pkts=23:15 dir=out,rev
2024-09-06T19:48:52 Notice kernel [lo=4183299327 high=4183363374 win=85 modulator=0 wscale=8] 9:7 R seq=4183299327 (4183299246) ack=1205303738 len=0 ackskew=0 pkts=22:15 dir=in,rev
2024-09-06T19:48:52 Notice kernel pf: loose state match: pf: loose state match: TCP out wire: 9.9.9.9TCP in wire: 192.168.1.191:46442 9.9.9.9:443 stack: - [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] [lo=4183299327 high=4183363374 win=85 modulator=0:443 MY_IP_ADDRESS:20913 stack: 9.9.9.9:443 192.168.1.191:46442 [lo=1205303738 high=1205324961 win=502 modulator=0 wscale=7] wscale=8] 9:7 R seq=1205303738 (1205303713) ack=4183299327 len=0 ackskew=0 pkts=22:15 dir=in,fwd
2024-09-06T19:44:04 Notice kernel pf: State failure on: |
2024-09-06T19:44:04 Notice kernel pf: BAD state: TCP out wire: 3.125.149.81:80 MY_IP_ADDRESS:31705 stack: 3.125.149.81:80 192.168.1.163:48580 [lo=3065242015 high=3065296889 win=332 modulator=0 wscale=8] [lo=4024237747 high=4024322676 win=490 modulator=0 wscale=7] 4:4 R seq=4024237747 (4024237684) ack=3065242015 len=0 ackskew=0 pkts=224:158 dir=in,rev
2024-09-06T19:41:12 Notice kernel pf: dropping packet with ip options
2024-09-06T19:39:45 Notice kernel pf: loose state match: TCP in wire: 192.168.1.148:61741 104.74.40.214:443 stack: - [lo=2066156501 high=2066220628 win=1026 modulator=0 wscale=8] [lo=272285939 high=272548595 win=502 modulator=0 wscale=7] 10:10 R seq=272285939 (272284883) ack=2066156501 len=0 ackskew=0 pkts=14:15 dir=out,rev
2024-09-06T19:39:45 Notice kernel pf: loose state match: TCP out wire: 104.74.40.214:443 MY_IP_ADDRESS:64218 stack: 104.74.40.214:443 192.168.1.148:61741 [lo=2066156501 high=2066220628 win=1026 modulator=0 wscale=8] [lo=272285939 high=272548595 win=502 modulator=0 wscale=7] 10:10 R seq=272285939 (272284883) ack=2066156501 len=0 ackskew=0 pkts=14:15 dir=in,rev
2024-09-06T19:34:32 Notice kernel pf: State failure on: |
2024-09-06T19:34:32 Notice kernel pf: BAD state: TCP out wire: 43.245.48.50:443 MY_IP_ADDRESS:12329 stack: 43.245.48.50:443 192.168.1.163:53592 [lo=3962970762 high=3963035247 win=414 modulator=0 wscale=8] [lo=3255328283 high=3255434056 win=510 modulator=0 wscale=7] 4:4 R seq=3255328283 (3255328072) ack=3962970762 len=0 ackskew=0 pkts=44:37 dir=in,rev
2024-09-06T19:33:59 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:59 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:6 dir=in,fwd
2024-09-06T19:33:48 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:48 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=21720 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:5 dir=in,fwd
2024-09-06T19:33:42 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:42 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:5 dir=in,fwd
2024-09-06T19:33:34 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:34 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:4 dir=in,fwd
2024-09-06T19:33:34 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:34 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=21720 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:4 dir=in,fwd
2024-09-06T19:33:30 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:30 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:3 dir=in,fwd
2024-09-06T19:33:30 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:30 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=21720 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:3 dir=in,fwd
2024-09-06T19:33:28 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:28 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=21720 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:2 dir=in,fwd
2024-09-06T19:33:28 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:28 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=21720 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:2 dir=in,fwd
2024-09-06T19:33:27 Notice kernel pf: State failure on: 4 |
2024-09-06T19:33:27 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065937 high=664087698 win=502 modulator=0 wscale=7] [lo=1081569324 high=1081633580 win=85 modulator=0 wscale=8] 4:2 A seq=664065937 (664065986) ack=665186288 len=0 ackskew=416383036 pkts=1:1 dir=in,fwd
2024-09-06T19:33:27 Notice kernel pf: State failure on: 3 |
2024-09-06T19:33:27 Notice kernel pf: BAD state: TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432148 high=1914453909 win=502 modulator=0 wscale=7] [lo=3747325126 high=3747389382 win=85 modulator=0 wscale=8] 4:2 A seq=1914432148 (1914432197) ack=1381881699 len=0 ackskew=-1929523869 pkts=1:1 dir=in,fwd
2024-09-06T19:33:27 Notice kernel pf: state reuse TCP out wire: 9.9.9.9:53 MY_IP_ADDRESS:62522 stack: 9.9.9.9:53 192.168.1.192:44770 [lo=664065986 high=664087746 win=502 modulator=0 wscale=7] [lo=665186288 high=665250416 win=85 modulator=0 wscale=8] 9:9 S
2024-09-06T19:33:27 Notice kernel pf: state reuse TCP in wire: 192.168.1.192:44770 9.9.9.9:53 stack: - [lo=664065986 high=664087746 win=502 modulator=0 wscale=7] [lo=665186288 high=665250416 win=85 modulator=0 wscale=8] 9:9 S
2024-09-06T19:33:27 Notice kernel pf: state reuse TCP out wire: 9.9.9.9:53 MY_IP_ADDRESS:41776 stack: 9.9.9.9:53 192.168.1.192:44762 [lo=1914432197 high=1914453957 win=502 modulator=0 wscale=7] [lo=1381881699 high=1381945827 win=85 modulator=0 wscale=8] 9:9 S
2024-09-06T19:33:27 Notice kernel pf: state reuse TCP in wire: 192.168.1.192:44762 9.9.9.9:53 stack: - [lo=1914432197 high=1914453957 win=502 modulator=0 wscale=7] [lo=1381881699 high=1381945827 win=85 modulator=0 wscale=8] 9:9 S
2024-09-06T19:33:23 Notice kernel pf: loose state match: TCP out wire: 172.64.150.233:443 MY_IP_ADDRESS:48942 stack: 172.64.150.233:443 192.168.1.191:37534 [lo=3240971823 high=3241045551 win=131 modulator=0 wscale=9] [lo=1631947932 high=1632014491 win=9 modulator=0 wscale=13] 9:7 R seq=3240971823 (3240971680) ack=1631947932 len=0 ackskew=0 pkts=6:5 dir=out,fwd
2024-09-06T19:33:23 Notice kernel pf: loose state match: TCP in wire: 192.168.1.191:37534 172.64.150.233:443 stack: - [lo=3240971823 high=3241045551 win=131 modulator=0 wscale=9] [lo=1631947932 high=1632014491 win=9 modulator=0 wscale=13] 9:7 R seq=3240971823 (3240971680) ack=1631947932 len=0 ackskew=0 pkts=6:5 dir=in,fwd
2024-09-06T19:33:20 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:24016 stack: 52.98.142.136:443 192.168.1.159:57876 [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:20 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57876 52.98.142.136:443 stack: - [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:8114 stack: 52.98.142.136:443 192.168.1.159:57876 [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57876 52.98.142.136:443 stack: - [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:56699 stack: 52.98.142.136:443 192.168.1.159:57875 [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57875 52.98.142.136:443 stack: - [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:49204 stack: 52.98.142.136:443 192.168.1.159:57876 [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57876 52.98.142.136:443 stack: - [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:61140 stack: 52.98.142.136:443 192.168.1.159:57875 [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:19 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57875 52.98.142.136:443 stack: - [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:41275 stack: 52.98.142.136:443 192.168.1.159:57876 [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57876 52.98.142.136:443 stack: - [lo=2563543327 high=2563551540 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:38905 stack: 52.98.142.136:443 192.168.1.159:57875 [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57875 52.98.142.136:443 stack: - [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP out wire: 52.98.142.136:443 MY_IP_ADDRESS:46832 stack: 52.98.142.136:443 192.168.1.159:57875 [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57875 52.98.142.136:443 stack: - [lo=1488153813 high=1488162026 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:28885 stack: 52.98.140.56:443 192.168.1.159:57874 [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:18 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57874 52.98.140.56:443 stack: - [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:27570 stack: 52.98.140.56:443 192.168.1.159:57874 [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57874 52.98.140.56:443 stack: - [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:11784 stack: 52.98.140.56:443 192.168.1.159:57873 [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57873 52.98.140.56:443 stack: - [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:17 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:57899 stack: 52.98.140.56:443 192.168.1.159:57874 [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57874 52.98.140.56:443 stack: - [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:25304 stack: 52.98.140.56:443 192.168.1.159:57873 [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57873 52.98.140.56:443 stack: - [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:1100 stack: 52.98.140.56:443 192.168.1.159:57874 [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57874 52.98.140.56:443 stack: - [lo=4194839826 high=4194848039 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:16579 stack: 52.98.140.56:443 192.168.1.159:57873 [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:16 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57873 52.98.140.56:443 stack: - [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP out wire: 52.98.140.56:443 MY_IP_ADDRESS:12072 stack: 52.98.140.56:443 192.168.1.159:57873 [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57873 52.98.140.56:443 stack: - [lo=3154300485 high=3154308698 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:59506 stack: 40.99.134.24:443 192.168.1.159:57872 [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57872 40.99.134.24:443 stack: - [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:46007 stack: 40.99.134.24:443 192.168.1.159:57872 [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:15 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57872 40.99.134.24:443 stack: - [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:37695 stack: 40.99.134.24:443 192.168.1.159:57871 [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57871 40.99.134.24:443 stack: - [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:55784 stack: 40.99.134.24:443 192.168.1.159:57872 [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57872 40.99.134.24:443 stack: - [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:12364 stack: 40.99.134.24:443 192.168.1.159:57871 [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:14 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57871 40.99.134.24:443 stack: - [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:45599 stack: 40.99.134.24:443 192.168.1.159:57872 [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57872 40.99.134.24:443 stack: - [lo=3657148362 high=3657156575 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:31168 stack: 40.99.134.24:443 192.168.1.159:57871 [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57871 40.99.134.24:443 stack: - [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP out wire: 40.99.134.24:443 MY_IP_ADDRESS:28098 stack: 40.99.134.24:443 192.168.1.159:57871 [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57871 40.99.134.24:443 stack: - [lo=2834883053 high=2834891266 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:18378 stack: 52.98.140.24:443 192.168.1.159:57870 [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57870 52.98.140.24:443 stack: - [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:13 Notice kernel pf: loose state match: TCP in wire: 192.168.1.159:57841 52.113.194.132:443 stack: - [lo=508273165 high=512467981 win=258 modulator=0 wscale=8] [lo=2514990012 high=2515056060 win=16386 modulator=0 wscale=8] 10:10 R seq=2514990012 (2514987340) ack=508273165 len=0 ackskew=0 pkts=35:71 dir=out,rev
2024-09-06T19:33:13 Notice kernel pf: loose state match: TCP out wire: 52.113.194.132:443 MY_IP_ADDRESS:42597 stack: 52.113.194.132:443 192.168.1.159:57841 [lo=508273165 high=512467981 win=258 modulator=0 wscale=8] [lo=2514990012 high=2515056060 win=16386 modulator=0 wscale=8] 10:10 R seq=2514990012 (2514987340) ack=508273165 len=0 ackskew=0 pkts=35:71 dir=in,rev
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:9686 stack: 52.98.140.24:443 192.168.1.159:57870 [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57870 52.98.140.24:443 stack: - [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:20532 stack: 52.98.140.24:443 192.168.1.159:57869 [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57869 52.98.140.24:443 stack: - [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:30982 stack: 52.98.140.24:443 192.168.1.159:57870 [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57870 52.98.140.24:443 stack: - [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:34651 stack: 52.98.140.24:443 192.168.1.159:57869 [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:12 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57869 52.98.140.24:443 stack: - [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:11 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:10533 stack: 52.98.140.24:443 192.168.1.159:57870 [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:11 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57870 52.98.140.24:443 stack: - [lo=3733468580 high=3733476793 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:11 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:4752 stack: 52.98.140.24:443 192.168.1.159:57869 [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:11 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57869 52.98.140.24:443 stack: - [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP out wire: 52.98.140.24:443 MY_IP_ADDRESS:65217 stack: 52.98.140.24:443 192.168.1.159:57869 [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57869 52.98.140.24:443 stack: - [lo=862971873 high=862980086 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:24747 stack: 52.98.143.136:443 192.168.1.159:57865 [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57865 52.98.143.136:443 stack: - [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:33258 stack: 52.98.143.136:443 192.168.1.159:57865 [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57865 52.98.143.136:443 stack: - [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:6125 stack: 52.98.143.136:443 192.168.1.159:57864 [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:10 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57864 52.98.143.136:443 stack: - [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:23050 stack: 52.98.143.136:443 192.168.1.159:57863 [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57863 52.98.143.136:443 stack: - [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:5126 stack: 52.98.143.136:443 192.168.1.159:57865 [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57865 52.98.143.136:443 stack: - [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:14239 stack: 52.98.143.136:443 192.168.1.159:57864 [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57864 52.98.143.136:443 stack: - [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:63286 stack: 52.98.143.136:443 192.168.1.159:57863 [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57863 52.98.143.136:443 stack: - [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:36042 stack: 52.98.143.136:443 192.168.1.159:57865 [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57865 52.98.143.136:443 stack: - [lo=428928225 high=428936438 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:25502 stack: 52.98.143.136:443 192.168.1.159:57864 [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:09 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57864 52.98.143.136:443 stack: - [lo=1040090196 high=1040098409 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:08 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:30954 stack: 52.98.143.136:443 192.168.1.159:57863 [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:08 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57863 52.98.143.136:443 stack: - [lo=119806819 high=119815032 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:08 Notice kernel pf: state reuse TCP out wire: 52.98.143.136:443 MY_IP_ADDRESS:13292 stack: 52.98.143.136:443 192.168.1.159:57862 [lo=3292658861 high=3292667074 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
2024-09-06T19:33:08 Notice kernel pf: state reuse TCP in wire: 192.168.1.159:57862 52.98.143.136:443 stack: - [lo=3292658861 high=3292667074 win=251 modulator=0] [lo=0 high=251 win=8212 modulator=0] 10:10 S
Maybe the excessive logging is causing the spikes. Also, I wonder if the dreaded FreeBSD SA causes this.
Could you try
opnsense-update -zkr 24.7.3-no_sa
and reboot?
Big excuse mes, but what's the FreeBSD SA?
I do notice I somehow turned on the additional logging at some point and cannot for the life of me work that out. It seems counterintuitive - logging (thing to find problems) causing problems (helped by logging)... Ha. Makes sense though - every action doubled.
Definitely can run that - can you explain what it does?
Thanks a bunch!
Oh also, worth noting I'm not on 24.7 as yet, still 24.1 - I saw the big thread about the IPv6/ICMP stuff (very fascinating, fwiw) and thought I'd avoid for a few weeks until some launch bugs were worked out. Is that what you mean? Would it matter on 24.1 Sav Shark?
No, I thought you were current.
I could do an upgrade if you think it might help out here?
No real need there, I just thought that maybe excessive logging of newly-introduced pf: messages created your CPU spikes. Im may still be the case, however, that kernel only eliminates a certain part of those messages. As I do not have logging for debug messages, I cannot say if those messages have increased and if so, when.
You could try to lower your logging to Debug: ...only for serious errors under Firewall: Settings: Advanced to try if you did not already have that.
Thank you @meyergru for your help.
I'm not sure if it fully helps, but I tried stopping the logging daemon entirely for a while - no change. I also stopped Suricata again but there really isn't anything else I can think of to stop that wouldn't compromise the device's functionality or security.
As an update, here's where things are up to. Over the weekend I took a clone of the disk (just for solid backup certainty - thinking I try the major upgrade and see if that somehow fixes this) and repasted the CPU. It wasn't showing extreme thermals, but not being familiar with the J3160, I wasn't sure if it perhaps had a TJM of 50/60 degrees instead of 90/100. It was preeeety crusty and dry, so that wasn't a bad shout. Unfortunately, no change.
My network has an NTD which terminates coax to Ethernet for the WAN, so I took the chance to reboot everything else too, including the switch, since sometimes some switches can get overloaded (and while it's not likely, the fact that I have similar dropping issues on both LAN [over EoP injectors] and WiFi could indiate the switch, since both go into that.)
I did try the old ISP router over dinner but I think I wired it wrong because it never came up, but I intend to test to see if my connection is solid on it as well. That would really narrow things down to either OPNsense or it's hardware. Beyond that... MAYBE a fresh install would prove it's this current firmware/config?
I'm starting to think that perhaps I'm just at the limit of this old thing's capabilities, for whatever reason. Perhaps single-core spikes throttle it or perhaps I have a bad binned chip... I can't tell. I've combed through OPNsense's settings a few times to see where something could have gone wrong, and short of rolling back another config to something, say, over a month old, I'm out of ideas.
Thanks again as always to everyone for helping me out here. Let me know if you have any last ideas!
I've upgraded to the latest version of OPNSense (24.7.3_1) successfully (side-note; props for the smooth upgrade process, OPNsense team!!) and will test tonight over the usual workloads. It's great to see more granular visuals and get a better at-a-glance understanding of system interrupts and temperatures here!
I will also test with the ISP router when I get the chance one of these nights - just snowed in with work OOO. If THAT fails, then I'm looking to grab one of the following devices as a replacement with the goal to upgrade to 2-4 Intel NICs and a display-supporting USB-C port (for easy portable screen console viewing instead of having to lug a whole display with power and HDMI up the freaking wall).
I can't dump enough money into this to go up to the ~$600-$800 that a decent Protectli device would do me but am trying to get as close as possible to that level.
HUNSN RH02
HUNSN JR03
HUNSN RJ12
TRIGKEY Green G5
Protectli Vault FW2B
Protectli Vault Pro VP2420
I would advise to get an N100-based device with 4x I226-V, HUNSN has those, too (e.g. HUNSN RJ36 (https://www.amazon.de/Firewall-Appliance-HUNSN-Barebone-Storage-N100/dp/B0BZJC12VP?ref_=ast_sto_dp&th=1&psc=1)) and they are also available from Kingnovy and Topton via Aliexpress and cost around the same as older CPU generations like N5000 or J4000.
Also, do not get one with the case of the devices you mentioned, as it has bad thermals. Maybe you must also tweak the TDP of the N100 a little down in the BIOS, because most of these devices use more than the Intel preset of 6 Watts.
Well, against all possible reason, it seems the upgrade has helped! It's early days, but so far the usual evening routine (discord chats, and played some Rocket League both as felt like it but also as a good test) and had no issues at all.
I'll continue to test for the next few days, and, all going well, update once more and call this one (thankfully, gratefully, welcomingly) done.
Thanks all for your help throughout.
Last update for this thread!
So far, all's going smoothly. I've actually been seeing very poor mobile performance for a long time on this OPNsense box--WiFiman tests frequently do 1-3mbps down and <1mbps up and then fail before finishing, on what should be a 50/20 link--until this update. Now I can actually do a test at ~30mbps down, ~12mbps up and complete the test. There MUST have just been something wrong with my initial setup or something screwed up over the months since implementing the device, which the new OS has overwritten.
I have one last, quick question if someone doesn't mind a brief answer! (I've taken up enough, haha!)
.4 has just released, but I'm a bit afraid to update again in case something regressed in this update somehow affects me. Would you recommend upgrading?
A big, final, thank you to everyone for your help in this thread, sincerely.
Well, the problem's back. I'm not going to drag this thread on any further, aside from updating it with key information for those googling around in the future who might stumble across it. No one needs to rush to responding to this.
At this point I'm just guessing it's the Realtek NICs. There's almost nothing else I can think of that could be it. The new Interrupts monitor in the 24.7 dashboard can now show me one of my last suspects--system interrupts--and unless it's ALSO hanging when interrupting (and so not drawing the graph) it's not that. My interrupts jump to 1/2 after a boot, then remain at 0 solidly forever after. My CPU usage and other monitoring shows only roughly 30%-60%utilisation in short spikes, so I don't think the J3160 is overloaded.
It was worth a try. I went into this knowing the public discourse around Realtek NICs, but I was encouraged by other users who'd had flawless functionality on them, and got a steal of a deal on this old dual-NIC box. Ultimately it wasn't to work out but at least I gave it a shot.
I'm going to aim for something like the HUNSN JR03 - I really, really want USB-C DP so I can use my little portable C-powered display, rather than lumping a whacking great monitor, power and display up to the half-depth shelf everything's tucked away neatly onto. It's tough to find the cash for it but here's hoping I can clear out some old crap - might even have to switch back to the ISP TP-Link for a few months until Black Friday sales hit.
Again, and finally, thank you to everyone for your help in this thread. Signing off. Safe networking to you all!