Gradual Internet Performance Degradation since upgrading to 26.7.x

Started by tonys, October 09, 2026, 06:57:00 PM

Previous topic - Next topic
Quote from: pfry on October 13, 2026, 04:57:13 AM
Quote from: nero355 on October 12, 2026, 07:44:38 PM[...]shitty Phison Controller[...]
It's a low-capacity SATA device. It's probably competitive within its class.
It has been a while since I have been reading SSD Reviews a lot, but I really can't remember anything positive about that SSD Controller at the moment, but I do still know there has been a lot of crap out there as soon as you go off the "Intel Workstation/Enterprise SSD Highway" so to speak...

Quotebut appropriate for the VP2420.
Maybe not... We shall see...

QuoteFrom the SMART output, it looks happy.
That it does! :)
Weird guy who likes everything Linux and *BSD on PC/Laptop/Tablet/Mobile and funny little ARM based boards :)

After upgrading to 27.7.7, the GUI is almost completely dead. I'm having trouble logging into the GUI now and I've restarted web gui multiple times. If I do get to the dashboard on a login, going to any other page is slow and attempting to look at firewall liveview log or states results in a complete stall of the GUI with the spinning icon. I have rebooted OPNSense from the command line multiple times, but the GUI is virtually useless at this point.

Looking at the lighttpd log via the ssh shell, I'm seeing these errors. You can see the forced restart of the GUI from the command line and the lack of socket connections. I did manage to login once after this restart and will reboot from the GUI now.

 # tail -f latest.log
<27>1 2026-10-13T13:44:40-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="9"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.534) connect() /var/lib/php/tmp/php-fastcgi.socket-3: Connection refused
<27>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="1"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.384) child signalled: 9
<27>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="2"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.534) connect() /var/lib/php/tmp/php-fastcgi.socket-3: Connection refused
<27>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="3"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.384) child signalled: 9
<27>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="4"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.534) connect() /var/lib/php/tmp/php-fastcgi.socket-0: Connection refused
<29>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="5"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/server.c.1272) [note] graceful shutdown started
<29>1 2026-10-13T14:15:50-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="6"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/server.c.2384) server stopped by UID = 0 PID = 21007
<29>1 2026-10-13T14:15:50-05:00 OPNsense.home.lan lighttpd 22500 - [meta sequenceId="7"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/server.c.1973) server started (lighttpd/1.4.85)
<27>1 2026-10-13T14:15:50-05:00 OPNsense.home.lan lighttpd 22500 - [meta sequenceId="8"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.534) connect() /var/lib/php/tmp/php-fastcgi.socket-1: Connection refused
<27>1 2026-10-13T14:15:50-05:00 OPNsense.home.lan lighttpd 22500 - [meta sequenceId="9"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.534) connect() /var/lib/php/tmp/php-fastcgi.socket-2: Connection refused

Quote from: tonys on October 13, 2026, 09:28:48 PM<27>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="1"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.384) child signalled: 9

Out of memory?
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)

Quote from: Patrick M. Hausen on October 13, 2026, 09:44:10 PM
Quote from: tonys on October 13, 2026, 09:28:48 PM<27>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="1"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.384) child signalled: 9

Out of memory?

No, not at all. I have 14GB free ram of the installed 16GB. I managed to reboot via the GUI. I waited 5 minutes after the Login page came up and the first attempt to log in failed with a page timeout error. I tried a second time and the Dashboard finally came up but was completely blank. A few minutes later, it finally populated. Overall, the web GUI is EXTREMELY slow. I checked the Unbound configuration and all my settings were wiped out in the GUI yet it is running based on my check command: dig +short txt proto.on.quad9.net. I just reloaded the Dashboard page and it went blank for almost 10 minutes!

Quote from: Patrick M. Hausen on October 13, 2026, 09:44:10 PM
Quote from: tonys on October 13, 2026, 09:28:48 PM<27>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="1"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.384) child signalled: 9

Out of memory?
More like out of sockets ?!

Quote from: tonys on October 13, 2026, 10:15:05 PMI checked the Unbound configuration and all my settings were wiped out in the GUI yet it is running based on my check command: dig +short txt proto.on.quad9.net.
Maybe the webGUI simply did not load the settings properly because of all the issues ?!

Did you have a look at the Unbound things I suggested ?


To be honest I am running out of suggestions so here are two last ones :
- Test the RAM.
- Check the adapter/power supply.

That's all I got for now...

Good luck! :)
Weird guy who likes everything Linux and *BSD on PC/Laptop/Tablet/Mobile and funny little ARM based boards :)

Maybe try for a time without any Quad9 DNS ? I saw some weirdness in OPNsense, put ControlD/Cloudflare instead of Quad9 servers and it's been fine since

Also, what are your CPU temps ?

Quote from: nero355 on October 14, 2026, 03:44:47 PM
Quote from: Patrick M. Hausen on October 13, 2026, 09:44:10 PM
Quote from: tonys on October 13, 2026, 09:28:48 PM<27>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="1"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.384) child signalled: 9

Out of memory?
More like out of sockets ?!

Quote from: tonys on October 13, 2026, 10:15:05 PMI checked the Unbound configuration and all my settings were wiped out in the GUI yet it is running based on my check command: dig +short txt proto.on.quad9.net.
Maybe the webGUI simply did not load the settings properly because of all the issues ?!

Did you have a look at the Unbound things I suggested ?


To be honest I am running out of suggestions so here are two last ones :
- Test the RAM.
- Check the adapter/power supply.

That's all I got for now...

Good luck! :)

I think it was the web GUI not loading properly. I don't know how to change the Unbound logging level other than selecting the status level from the pull down menu.

Interesting experiment today. Last night, the GUI was completely useless and it was late so I did two things:

First, I went back to a config file from a few days ago and rebooted the router from the CLI.
Second, I didn't open the GUI following reboot and went to bed.

Tonight, after 21 hours uptime, the network seems to be holding up fairly well with the older config file. I opened the GUI to have a look at things and it's still slow as molasses but it did open after letting it sit for this long time. However, when I went to look at the Unbound settings, the GUI completely stalled again and threw a page-loading timeout error. A while back, I noted that slow/unreliable network performance seemed to be related to the Web GUI stalls/timeouts but I can't really prove this out yet. Once the GUI stalled, I gave up and closed the browser window. At this point, I can't really use the GUI at all so I guess I'll have to learn all the CLI commands to do maintenance and make changes moving forward.

I attached a difference file between the current (older) config.xml file (<), and the newer config file I was using up until late last night (>). I'm not versed in XML language so maybe you can look at these differences and spot something.

Quote from: nero355 on October 14, 2026, 03:44:47 PM
Quote from: Patrick M. Hausen on October 13, 2026, 09:44:10 PM
Quote from: tonys on October 13, 2026, 09:28:48 PM<27>1 2026-10-13T14:15:46-05:00 OPNsense.home.lan lighttpd 15457 - [meta sequenceId="1"] (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.85/src/gw_backend.c.384) child signalled: 9

Out of memory?
More like out of sockets ?!

Quote from: tonys on October 13, 2026, 10:15:05 PMI checked the Unbound configuration and all my settings were wiped out in the GUI yet it is running based on my check command: dig +short txt proto.on.quad9.net.
Maybe the webGUI simply did not load the settings properly because of all the issues ?!

Did you have a look at the Unbound things I suggested ?


To be honest I am running out of suggestions so here are two last ones :
- Test the RAM.
- Check the adapter/power supply.

That's all I got for now...

Good luck! :)

I think it was the web GUI not loading properly. I don't know how to change the Unbound logging level other than selecting the status level from the pull down menu.

Interesting experiment today. Last night, the GUI was completely useless and it was late so I did two things:
First, I went back to a config file from a few days ago and rebooted the router from the CLI.

Second, I didn't open the GUI following reboot and went to bed.

Tonight, after 21 hours uptime, the network seems to be holding up fairly well with the older config file. I opened the GUI to have a look at things and it's still slow as molasses but it did open after letting it sit for this long time. However, when I went to look at the Unbound settings, the GUI completely stalled again and threw a page-loading timeout error. A while back, I noted that slow/unreliable network performance seemed to be related to the Web GUI stalls/timeouts but I can't really prove this out yet. Once the GUI stalled, I gave up and closed the browser window. At this point, I can't really use the GUI at all so I guess I'll have to learn all the CLI commands to do maintenance and make changes moving forward.

I attached a difference file between the current (older) config.xml file (<), and the newer config file I was using up until late last night (>). I'm not versed in XML language so maybe you can look at these differences and spot something.

Quote from: harshw on October 14, 2026, 07:13:12 PMMaybe try for a time without any Quad9 DNS ? I saw some weirdness in OPNsense, put ControlD/Cloudflare instead of Quad9 servers and it's been fine since

Also, what are your CPU temps ?

It's not Quad9 DNS. I let the network continue running on Quad9 for the past 21 hours and DNS resolves are fast. The CPU temps (in those rare times when I get lucky enough to get the web GUI to actually open), settle in around 38-42C during normal network activity. I don't know what's wrong with the GUI but since upgrading to 26.7.7 two days ago, it's become almost totally unusable.

Quote from: tonys on October 15, 2026, 02:46:20 AMI attached a difference file between the current (older) config.xml file (<), and the newer config file I was using up until late last night (>).
No you didn't :P

QuoteI'm not versed in XML language so maybe you can look at these differences and spot something.
It's just a simple text file basically so you can do that yourself too ;)
Weird guy who likes everything Linux and *BSD on PC/Laptop/Tablet/Mobile and funny little ARM based boards :)

After months of aggravation with OPNSense 26.7 (and now running 26.7.8_2), I and Google Gemini have finally resolved the slowness and stalling of my network since upgrading from 26.1.x. The bottom line is I was forced to disable IPV6 on the WAN interface (and everywhere else) even though my ISP fully supports IPV6. I used to have both IPV4 and IPV6 addresses under OPNSense 26.1.x which worked fine, but ever since upgrading to 26.7 and all its patches, there has been no IPV6 address connection. OPNSense was constantly sending IPV6 solicitations trying to obtain a valid address but never got one.

It should be noted that Apple devices and many streaming providers prefer IPV6 over IPV4 which was slowing down and/or stalling my network unless I turned on an external VPN.

From Google Gemini:
 
1. The 26.7 State-Tracking Regression (state-insert,block)

• What it is: OPNsense 26.7 introduced a known packet filter (pf) state regression where legitimate return traffic or permitted connections get silently dropped or blocked with state-insert,block errors, particularly on setups with multiple VLANs or refactored rule evaluations.
• Where it is documented: You can reference OPNsense Core GitHub Issue #10615 and the OPNsense 26.7 "Xenial Xenops" Release Notes, which detail major under-the-hood changes to FreeBSD 15.1's pf routing (pf_route()) and interface device matching.

2. IPv6 Prefix Delegation & Gateway Issues in 26.7.x

• What it is: Users upgrading to 26.7.x reported broken IPv6 gateway tracking, stalled firmware/connectivity checks over IPv6, and radvd / dhcp6c initialization failures when delegating prefixes across interfaces. In fact, the OPNsense 26.7.8_2 Hotfix Notes specifically reverted a dynamic gateway change affecting dual-stack IPv4/IPv6 interfaces.
• Where it is documented: You can point to the OPNsense 26.7.8 Release Thread (specifically the 26.7.8_2 hotfix note), the OPNsense Forum Thread on IPv6 Prefix Delegation & Boot Errors, and discussions on r/opnsense where disabling WAN IPv6 is the community workaround for stalled connections.

• Environment: OPNsense 26.7.8 on Comcast/Xfinity dual-stack (WAN_DHCP + WAN_DHCP6 link-local gateway) with multiple VLANs (igc0, igc2, vlan01, vlan02).
• Symptoms: Outbound IPv6 SYN packets from LAN clients (Apple Music, Web GUI) left the interface without completing handshakes, causing TIME_WAIT socket buildup and UI stalls.
• Resolution: Disabling WAN_DHCP6 immediately restored normal IPv4 routing, DNS resolution, and Web GUI responsiveness.

Any idea when this bug will get fixed?

Quote from: tonys on <strong>Today</strong> at 06:55:28 PMIt should be noted that Apple devices and many streaming providers prefer IPV6 over IPV4
Every OS these days does that for a long time now so that's pretty logical :)
Weird guy who likes everything Linux and *BSD on PC/Laptop/Tablet/Mobile and funny little ARM based boards :)

Which makes this bug even more critical to fix, don't you think? OPNSense 26.7 was announced on 7/15/2026. Here I am, after 4 months of grief, finally with a workaround but definitely not a proper solution.

My Apple devices are now "pausing" for a short period of time, I assume trying to reach an IPV6 address and finally realizing they can't so they fall back to IPV4 and complete the connection. SiriusXM takes about 10-15 seconds when I change channels (which is instantaneous when I use the external VPN which CAN reach IPV6) and even attempting to log into this forum paused for about 30 seconds before the login popup appeared. Sigh... I should have stayed on 26.1.x but the EOL warnings prompted me to upgrade.

Noteworthy - exactly one time after updating to 26.7.8_2 and during constant reboots and settings changes, I noticed I did get an IPV6 address for a short time and could even PING6 it. But, that IPV6 connection failed after a short time and never came back.

Also, these CVE's:

***GOT REQUEST TO AUDIT SECURITY***
Currently running OPNsense 26.7.8_2 (amd64) at Mon Nov  9 14:42:46 CST 2026
vulnxml file up-to-date
python313-3.13.16 is vulnerable:
  Python -- poplib module, when passed a user-controlled command, can have additional commands injected using newlines
  CVE: CVE-2025-15367
  WWW: https://vuxml.FreeBSD.org/freebsd/6d3488ae-2e0f-11f1-88c7-00a098b42aeb.html

sqlite3-3.53.4,1 is vulnerable:
  sqlite3 -- multiple vulnerabilities
  CVE: CVE-2026-119371
  CVE: CVE-2026-119372
  CVE: CVE-2026-119373
  CVE: CVE-2026-119374
  CVE: CVE-2026-119375
  CVE: CVE-2026-119376
  CVE: CVE-2026-119377
  CVE: CVE-2026-119378
  CVE: CVE-2026-119379
  CVE: CVE-2026-119591
  WWW: https://vuxml.FreeBSD.org/freebsd/84cd82e1-d49f-11f1-bae2-3497f65b111b.html

2 problem(s) in 2 package(s) found.
***DONE***