Gradual Internet Performance Degradation since upgrading to 26.7.x

[tonys]

Ever since upgrading from 21.x to 26.7.x, I've been experiencing gradually deteriorating internet performance. Specifically, after a router reboot, internet performance is great but over a period of a few hours it begins slowing down. LAN devices have trouble accessing the internet and at times, they can't connect at all. The OPNSense web GUI also slows down simultaneously with the internet slowdown to the point where I can't open the GUI at all. Typically, I log into an SSH shell and restart the web gui which sometimes helps but not always. Eventually, the only solution is a complete reboot of the router which then brings performance back to normal for a few hours and this cycle repeats.

The temporary fix I've found so far is to enable Proton VPN on the affected LAN devices which restores internet performance but does nothing for the OPNSense web GUI. After sitting overnight, the web GUI becomes somewhat responsive again but still slow and gradually degrades again after a few hours. Unfortunately, internet performance remains severely degraded and almost useless without the external VPN turned on. Rebooting the router restores LAN internet performance but only for a few hours.

I've noticed posts here about similar issues with earlier OPNSense releases after upgrades but haven't seen anything about this happening after upgrading from the final 21.x release to  26.7.x. Is anyone else experiencing this?

I'm currently running OPNSense 26.7.6 on a Protectli VP2420 with 16GB of ram. There are no OPNSense processes hogging CPU time, free ram is running about 13GB, the firewall state table hovers in the low 300's, and unnecessary services (ClamAV, Netflow) are turned off which helps a bit but not permanently. I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

Gemini and Grok recommend I wipe the Protectli flash and install OPNSense 26.7 from scratch, and then restore the router from the backed up config.xml but I don't know if this will work. Advice would be appreciated as I've been battling this slowly degrading performance cycle for a couple of months.

[nero355]

There are no OPNSense processes hogging CPU time

Post your 'top' output anyway :
- One after reboot.
- One when everything has slowed down.

And if you haven't already : Disable HostWatch

I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

A while ago someone had issues because the Unbound Logging Level was set too high and the SSD could not keep up with it IIRC so you could take a look at that too!

[tonys]

There are no OPNSense processes hogging CPU time

Post your 'top' output anyway :
- One after reboot.
- One when everything has slowed down.

And if you haven't already : Disable HostWatch

I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

A while ago someone had issues because the Unbound Logging Level was set too high and the SSD could not keep up with it IIRC so you could take a look at that too!

Ok, first things first. I only want to change one thing at a time to ease troubleshooting. The network slowed to a crawl again so I'm attaching a process dump file immediately after a reboot and one that took about a day and a half to reach enough slowness to cause me to do another reboot. I haven't changed any settings yet as Hostwatch was using almost 0 time and occupying only 16M of memory.

[nero355]

I only want to change one thing at a time to ease troubleshooting. The network slowed to a crawl again so I'm attaching a process dump file immediately after a reboot and one that took about a day and a half to reach enough slowness to cause me to do another reboot.

I don't see anything weird either so it got me thinking : Maybe it's the storage ?!

You also did not mention anything about the Unbound logging level setting that I mentioned earlier ?

Can you post the output of :

Code Select Expand

# smartctl /dev/<whatever you HDD/SSD is> -aand if possible also

Code Select Expand

# iotop Hopefully that's it and replacing the HDD/SSD will be enough to fix this...

[cookiemonster]

I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

Are you using blocklists with Unbound? Try without them. Monitor Unbound with logs, ps, etc.
Although is best to find the hogging process or activity and zoom in, that's what I would do if Unbound appears fine. We're only following a hunch after all.

and unnecessary services (ClamAV, Netflow) are turned off

Good. These are relatively heavy processes.

[tonys]

I only want to change one thing at a time to ease troubleshooting. The network slowed to a crawl again so I'm attaching a process dump file immediately after a reboot and one that took about a day and a half to reach enough slowness to cause me to do another reboot.

I don't see anything weird either so it got me thinking : Maybe it's the storage ?!

You also did not mention anything about the Unbound logging level setting that I mentioned earlier ?

Can you post the output of :

Code Select Expand

# smartctl /dev/<whatever you HDD/SSD is> -aand if possible also

Code Select Expand

# iotop Hopefully that's it and replacing the HDD/SSD will be enough to fix this...

Please note that /var and /tmp are running in a ramdisk. I moved to ramdisk a couple of weeks ago thinking the SSD was too slow but now I doubt it.

Attached is the smartctl output. Also note there is no ioctl command on my system?? Unbound logging level is set at error. Switching to debug mode, all I see are information and notices, no warnings and no errors.

[tonys]

I suspect Unbound is the culprit since my IoT and Guest networks bypass Unbound and run fine with DNS servers 8.8.8.8 and 1.1.1.1. Only the LAN is affected because Unbound passes its traffic to Quad9 servers. Unbound logs show all DNS traffic going out since I added bypasses from servers blocked by Steven's Blacklist.

Are you using blocklists with Unbound? Try without them. Monitor Unbound with logs, ps, etc.
Although is best to find the hogging process or activity and zoom in, that's what I would do if Unbound appears fine. We're only following a hunch after all.

and unnecessary services (ClamAV, Netflow) are turned off

Good. These are relatively heavy processes.

Yes, I'm using Stevens Blacklist with about a dozen domains whitelisted that affected the Mac laptops and phones. No unusual blocks being seen in the Unbound logs.

Most of my settings are the same as they were in the final release of 21.x. I had absolutely no problems at all under 21.x, all of my problems started after upgrading to the first and subsequent releases of 26.7. I have a feeling that all of this is related to Unbound targeting quad9 DNS servers because a key point is that turning on Proton VPN in any LAN device instantly restores fast connectivity to the internet. Turning Proton off causes stalls on some sites which I can sometimes get to by reloading the page. The browsers constantly throw "page took too long to load" errors regardless of the browser I'm using.

The OPNSense web gui is very slow to load at all times, even after a reboot. Logins can take 30-60 seconds and changing pages in the GUI is also very slow. After a day or two of running, I can't even get into the GUI and typically reboot the router at the command line. External web sites may or may not load immediately after a reboot so I turn on Proton. After reboot today, I couldn't get to this site without turning Proton on.

I've now turned off Stevens Blacklist and will test under these conditions. Unbound continues to route to Quad DNS.

[nero355]

Attached is the smartctl output.

Looks clean, but these are unknown :

Code Select Expand

170 Unknown_Attribute       0x0003   100   100   010    Pre-fail  Always       -       171
173 Unknown_Attribute       0x0012   100   100   000    Old_age   Always       -       7930004And since it seems to be a SSD rebranded by Protectli I can't find any manufacturer documentation about it...

Also note there is no ioctl command on my system??

iotop is a command for checking storage activity but maybe OPNsense does not have it by default.

Unbound logging level is set at error.

What happens when you completely disable it ?

Switching to debug mode, all I see are information and notices, no warnings and no errors.

That's good, but I think disabling it should avoid the storage issues that I have mentioned before!

Yes, I'm using Stevens Blacklist with about a dozen domains whitelisted that affected the Mac laptops and phones.
No unusual blocks being seen in the Unbound logs.

Most of my settings are the same as they were in the final release of 21.x. I had absolutely no problems at all under 21.x, all of my problems started after upgrading to the first and subsequent releases of 26.7.

I have a feeling that all of this is related to Unbound targeting quad9 DNS servers because a key point is that turning on Proton VPN in any LAN device instantly restores fast connectivity to the internet. Turning Proton off causes stalls on some sites which I can sometimes get to by reloading the page. The browsers constantly throw "page took too long to load" errors regardless of the browser I'm using.

The OPNSense web gui is very slow to load at all times, even after a reboot. Logins can take 30-60 seconds and changing pages in the GUI is also very slow. After a day or two of running, I can't even get into the GUI and typically reboot the router at the command line. External web sites may or may not load immediately after a reboot so I turn on Proton. After reboot today, I couldn't get to this site without turning Proton on.

I've now turned off Stevens Blacklist and will test under these conditions. Unbound continues to route to Quad DNS.

There was a topic a while ago that mentioned something about the way Unbound does the processing of the blocklist and how inefficient it is compared to let's say Pi-Hole and AdGuard.

Since I am a Pi-Hole user and follow their Discourse Forum frequently I have read something about the whole processing flow of the blocklists/whitelists not too long ago and after comparing it to the Unbound version it once again proved that Unbound is simply the wrong choice for network-wide adblocking functionality :)

So I would say consider leaving Unbound for what it is : A Recursive Resolver that only talks to the Root DNS Servers.
And use something else for adblocking...

[Patrick M. Hausen]

iotop is a command for checking storage activity but maybe OPNsense does not have it by default.

Use

Code Select Expand

iostat 2

instead, or if running on ZFS

Code Select Expand

zpool iostat 2


[tonys]

iotop is a command for checking storage activity but maybe OPNsense does not have it by default.

Use

Code Select Expand

iostat 2

instead, of if running on ZFS

Code Select Expand

zpool iostat 2


 # iostat 2
       tty           mmcsd0           mmcsd0           mmcsd0             cpu
 tin  tout KB/t   tps  MB/s KB/t   tps  MB/s KB/t   tps  MB/s  us ni sy in id
   0     3  3.2     0  0.00  2.9     0  0.00  2.9     0  0.00   1  0  0  0 99
   0   118  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  0  0 99
   0    39  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  0  0 100
   0    41  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   5  0  2  0 93
   0    39  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  0  0 100
   0    41  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  0  0 100
   0    39  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  1  0 99
   0    40  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  1  0 99
   0    38  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   5  0  2  0 93
   0    42  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  0  0 100
   0    40  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  0  0 100
   0    40  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  0  0 100
   3   269  0.0     0  0.00  0.0     0  0.00  0.0     0  0.00   0  0  0  0 100

[tonys]

Attached is the smartctl output.

Looks clean, but these are unknown :

Code Select Expand

170 Unknown_Attribute       0x0003   100   100   010    Pre-fail  Always       -       171
173 Unknown_Attribute       0x0012   100   100   000    Old_age   Always       -       7930004And since it seems to be a SSD rebranded by Protectli I can't find any manufacturer documentation about it...

Also note there is no ioctl command on my system??

iotop is a command for checking storage activity but maybe OPNsense does not have it by default.

Unbound logging level is set at error.

What happens when you completely disable it ?

Switching to debug mode, all I see are information and notices, no warnings and no errors.

That's good, but I think disabling it should avoid the storage issues that I have mentioned before!

Yes, I'm using Stevens Blacklist with about a dozen domains whitelisted that affected the Mac laptops and phones.
No unusual blocks being seen in the Unbound logs.

Most of my settings are the same as they were in the final release of 21.x. I had absolutely no problems at all under 21.x, all of my problems started after upgrading to the first and subsequent releases of 26.7.

I have a feeling that all of this is related to Unbound targeting quad9 DNS servers because a key point is that turning on Proton VPN in any LAN device instantly restores fast connectivity to the internet. Turning Proton off causes stalls on some sites which I can sometimes get to by reloading the page. The browsers constantly throw "page took too long to load" errors regardless of the browser I'm using.

The OPNSense web gui is very slow to load at all times, even after a reboot. Logins can take 30-60 seconds and changing pages in the GUI is also very slow. After a day or two of running, I can't even get into the GUI and typically reboot the router at the command line. External web sites may or may not load immediately after a reboot so I turn on Proton. After reboot today, I couldn't get to this site without turning Proton on.

I've now turned off Stevens Blacklist and will test under these conditions. Unbound continues to route to Quad DNS.

There was a topic a while ago that mentioned something about the way Unbound does the processing of the blocklist and how inefficient it is compared to let's say Pi-Hole and AdGuard.

Since I am a Pi-Hole user and follow their Discourse Forum frequently I have read something about the whole processing flow of the blocklists/whitelists not too long ago and after comparing it to the Unbound version it once again proved that Unbound is simply the wrong choice for network-wide adblocking functionality :)

So I would say consider leaving Unbound for what it is : A Recursive Resolver that only talks to the Root DNS Servers.
And use something else for adblocking...

This is interesting. In OPNSense 21.x, I had both Stevens Blacklist and Adguard turned on and never saw any network stalls or slow performance. I rarely even logged into OPNSense because I had no reason to.

So after 21 hours since the last reboot, with the blacklist disabled, I'm noticing network slowdowns and stalls again. Additionally, the OPNSense GUI is slowing down again but not nearly as bad as with the blacklist enabled. Turning on Proton VPN restores the network to normal speed. From what I've been reading, Unbound is bypassed when a device is using an external encrypted VPN so this seems to make sense. Another clue is that my IoT and Guest VLANS use public DNS - only the LAN uses encrypted Unbound/Quad9 - and I haven't seen any slowdowns on the IoT or Guest VLANS.

I'm going to run for another day without rebooting the router and see if the LAN degrades further.

[nero355]

I'm going to run for another day without rebooting the router and see if the LAN degrades further.

Good luck! :)

By the way this was the topic about Unbound a while ago : https://forum.opnsense.org/index.php?topic=52429.msg273588#msg273588

iotop is a command for checking storage activity but maybe OPNsense does not have it by default.

Use

Code Select Expand

iostat 2

instead, of if running on ZFS

Code Select Expand

zpool iostat 2


Thanks! :)

[pfry]

A bit of an aside:

[...]And since it seems to be a SSD rebranded by Protectli I can't find any manufacturer documentation about it...[...]

Apparently Smartbuy (from the firmware version)... go figure. Basic SATA SSD, Phison controller.

My Intel SATA SSDs from 2016 or so have mostly unknown SMART attributes as well. Some are identifiable by value. I have older devices with more complete and readable SMART attributes.

[nero355]

Apparently Smartbuy (from the firmware version)... go figure. Basic SATA SSD, Phison controller.

W-T-F ?!?! o.O

So the SSD is made and sold rebranded by some webshop in Russia with a shitty Phison Controller ?!

That's just ridiculous...



There was some guy here claiming how high quality Protectli devices are and super reliable and all that, but when I see all the issues posted here lately I am having some serious doubts about those claims !!! :(