Gradual Internet Performance Degradation since upgrading to 26.7.x

[tonys]

Ever since upgrading from 21.x to 26.7.x, I've been experiencing gradually deteriorating internet performance. Specifically, after a router reboot, internet performance is great but over a period of a few hours it begins slowing down. LAN devices have trouble accessing the internet and at times, they can't connect at all. The OPNSense web GUI also slows down simultaneously with the internet slowdown to the point where I can't open the GUI at all. Typically, I log into an SSH shell and restart the web gui which sometimes helps but not always. Eventually, the only solution is a complete reboot of the router which then brings performance back to normal for a few hours and this cycle repeats.

The temporary fix I've found so far is to enable Proton VPN on the affected LAN devices which restores internet performance but does nothing for the OPNSense web GUI. After sitting overnight, the web GUI becomes somewhat responsive again but still slow and gradually degrades again after a few hours. Unfortunately, internet performance remains severely degraded and almost useless without the external VPN turned on. Rebooting the router restores LAN internet performance but only for a few hours.

I've noticed posts here about similar issues with earlier OPNSense releases after upgrades but haven't seen anything about this happening after upgrading from the final 21.x release to  26.7.x. Is anyone else experiencing this?

I'm currently running OPNSense 26.7.6 on a Protectli VP2420 with 16GB of ram. There are no OPNSense processes hogging CPU time, free ram is running about 13GB, the firewall state table hovers in the low 300's, and unnecessary services (ClamAV, Netflow) are turned off which helps a bit but not permanently. I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

Gemini and Grok recommend I wipe the Protectli flash and install OPNSense 26.7 from scratch, and then restore the router from the backed up config.xml but I don't know if this will work. Advice would be appreciated as I've been battling this slowly degrading performance cycle for a couple of months.

[nero355]

There are no OPNSense processes hogging CPU time

Post your 'top' output anyway :
- One after reboot.
- One when everything has slowed down.

And if you haven't already : Disable HostWatch

I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

A while ago someone had issues because the Unbound Logging Level was set too high and the SSD could not keep up with it IIRC so you could take a look at that too!

[tonys]

There are no OPNSense processes hogging CPU time

Post your 'top' output anyway :
- One after reboot.
- One when everything has slowed down.

And if you haven't already : Disable HostWatch

I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

A while ago someone had issues because the Unbound Logging Level was set too high and the SSD could not keep up with it IIRC so you could take a look at that too!

Ok, first things first. I only want to change one thing at a time to ease troubleshooting. The network slowed to a crawl again so I'm attaching a process dump file immediately after a reboot and one that took about a day and a half to reach enough slowness to cause me to do another reboot. I haven't changed any settings yet as Hostwatch was using almost 0 time and occupying only 16M of memory.

[nero355]

I only want to change one thing at a time to ease troubleshooting. The network slowed to a crawl again so I'm attaching a process dump file immediately after a reboot and one that took about a day and a half to reach enough slowness to cause me to do another reboot.

I don't see anything weird either so it got me thinking : Maybe it's the storage ?!

You also did not mention anything about the Unbound logging level setting that I mentioned earlier ?

Can you post the output of :

Code Select Expand

# smartctl /dev/<whatever you HDD/SSD is> -aand if possible also

Code Select Expand

# iotop Hopefully that's it and replacing the HDD/SSD will be enough to fix this...

[cookiemonster]

I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

Are you using blocklists with Unbound? Try without them. Monitor Unbound with logs, ps, etc.
Although is best to find the hogging process or activity and zoom in, that's what I would do if Unbound appears fine. We're only following a hunch after all.

and unnecessary services (ClamAV, Netflow) are turned off

Good. These are relatively heavy processes.

[tonys]

I only want to change one thing at a time to ease troubleshooting. The network slowed to a crawl again so I'm attaching a process dump file immediately after a reboot and one that took about a day and a half to reach enough slowness to cause me to do another reboot.

I don't see anything weird either so it got me thinking : Maybe it's the storage ?!

You also did not mention anything about the Unbound logging level setting that I mentioned earlier ?

Can you post the output of :

Code Select Expand

# smartctl /dev/<whatever you HDD/SSD is> -aand if possible also

Code Select Expand

# iotop Hopefully that's it and replacing the HDD/SSD will be enough to fix this...

Please note that /var and /tmp are running in a ramdisk. I moved to ramdisk a couple of weeks ago thinking the SSD was too slow but now I doubt it.

Attached is the smartctl output. Also note there is no ioctl command on my system?? Unbound logging level is set at error. Switching to debug mode, all I see are information and notices, no warnings and no errors.

[tonys]

I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

Are you using blocklists with Unbound? Try without them. Monitor Unbound with logs, ps, etc.
Although is best to find the hogging process or activity and zoom in, that's what I would do if Unbound appears fine. We're only following a hunch after all.

and unnecessary services (ClamAV, Netflow) are turned off

Good. These are relatively heavy processes.

Yes, I'm using Stevens Blacklist with about a dozen domains whitelisted that affected the Mac laptops and phones. No unusual blocks being seen in the Unbound logs.

Most of my settings are the same as they were in the final release of 21.x. I had absolutely no problems at all under 21.x, all of my problems started after upgrading to the first and subsequent releases of 26.7. I have a feeling that all of this is related to Unbound targeting quad9 DNS servers because a key point is that turning on Proton VPN in any LAN device instantly restores fast connectivity to the internet. Turning Proton off causes stalls on some sites which I can sometimes get to by reloading the page. The browsers constantly throw "page took too long to load" errors regardless of the browser I'm using.

The OPNSense web gui is very slow to load at all times, even after a reboot. Logins can take 30-60 seconds and changing pages in the GUI is also very slow. After a day or two of running, I can't even get into the GUI and typically reboot the router at the command line. External web sites may or may not load immediately after a reboot so I turn on Proton. After reboot today, I couldn't get to this site without turning Proton on.

I've now turned off Stevens Blacklist and will test under these conditions. Unbound continues to route to Quad DNS.