Hi all,
after upgrade from 25.7.11_2 to 26.1r1 everything looked good at first. I did not yet try the rule migration but intended to wait for RC2 with all the fixes in that specific area.
Half an hour later Internet was down. SSH to the box still working, system quite sluggish, dashboard widgets failing to load.
A couple of hundred processes like this:
/usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe0 force
"killall -9 php" made the system responsive again for a short while but the processes kept piling up.
Anything specific in the log I should look for?
With 25.7 running, this is the dhcp6d.conf:
interface pppoe0 {
send ia-na 2; # request stateful address
send ia-pd 2; # request prefix delegation
request domain-name-servers;
request domain-name;
script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
id-assoc na 2 { };
id-assoc pd 2 {
prefix ::/56 infinity;
};
I'm a bit puzzled by that "request domain-name-servers;" - is that hard coded? I could not find a way to disable it, anywhere and I certainly do not want any DNS servers, be it v4 or v6 from my ISP.
I isolated the logs for a single PID when the system was running 26.1r1:
root@opnsense:/var/log/system # grep 56100 *
latest.log:<29>1 2026-01-25T14:53:57+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="701"] Sending Solicit
latest.log:<27>1 2026-01-25T14:53:57+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="702"] transmit failed: Can't assign requested address
latest.log:<29>1 2026-01-25T14:53:58+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="705"] Sending Solicit
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="721"] Sending Request
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="722"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="723"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="724"] failed to update an address ::
latest.log:<29>1 2026-01-25T14:54:00+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="732"] Sending Solicit
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="733"] Sending Request
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="734"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="735"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="736"] failed to update an address ::
latest.log:<29>1 2026-01-25T14:54:02+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="741"] Sending Solicit
latest.log:<29>1 2026-01-25T14:54:03+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="744"] Sending Request
latest.log:<29>1 2026-01-25T14:54:06+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="746"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:54:06+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="747"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:54:06+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="748"] failed to update an address ::
latest.log:<29>1 2026-01-25T14:54:07+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="757"] Sending Solicit
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="759"] Sending Request
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="760"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="761"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="762"] failed to update an address ::
[...]
system_20260125.log:<29>1 2026-01-25T16:12:17+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20634"] Sending Solicit
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20640"] Sending Request
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20641"] Received REPLY for REQUEST
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20642"] failed to remove an address on pppoe0: Can't assign requested address
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20643"] failed to update an address ::
system_20260125.log:<29>1 2026-01-25T16:12:19+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20646"] Sending Solicit
root@opnsense:/var/log/system #
Kind regards,
Patrick
Quote from: Patrick M. Hausen on January 25, 2026, 05:55:15 PMHalf an hour later Internet was down. SSH to the box still working, system quite sluggish, dashboard widgets failing to load.
A couple of hundred processes like this:
/usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe0 force
"killall -9 php" made the system responsive again for a short while but the processes kept piling up.
Sounds similar to : https://forum.opnsense.org/index.php?msg=257256 ??
I don't use Suricata or Zenarmor.
https://forum.opnsense.org/index.php?topic=50500.0 appears to be the same and I've heard that privately before once too.
The questions are these:
1. Did you configure anything else than what 25.7.x already offered?
2. Do you have the full system log around the incident to inspect?
It looks like a loop we've seen many years ago in PPPoE vs. rc.newwanipv6 script and there haven't been much changes. I'm taking an educated guess with a commit in a second we can easily flush to 26.1-RC2.
Cheers,
Franco
I uninstalled the ISC DHCP plugin, but did not change anything else.
I'll send you an email.
Thanks!
Patrick
Thanks, that would further fuel the suspicion. I'll want to add https://github.com/opnsense/core/commit/3248b4d2315f to RC2 and see how it goes (with extra logging).
Going through your log in a second as well and reporting back.
Cheers,
Franco
Situation unchanged with RC2. Already sent a debug log to Franco.
# opnsense-patch https://github.com/opnsense/core/commit/9a80c6ddb29
This should make it behave as before on 25.7.x, best done with a reboot right after apply.
If that's the case we found a very old bug in dhcp6c.
Cheers,
Franco
I already had a complete post telling you "nope", but then decided to give it one more try, because with your hint in the private email I had enabled rapid commit.
So the results so far:
RC2 with the patch, same configuration as with 25.7: fine
RC2 with the patch, rapid commit enabled: dpcp6c processes spawning again
I'll leave the patched RC2 running for now.
HTH,
Patrick
P.S. Why is the just booted system with no UI session active running 25 php-cgi processes?
Rapid-commit (new in 26.1) may run into the same issue if the server refused the request altogether. It's probably going to remain optional anyway. Let he hotfix that one then.
About php-cgi... this is normal
root 82341 0.0 0.1 23728 11060 - S 13:50 0:01.69 |-- /usr/local/sbin/lighttpd -f /usr/local/etc/lighttpd_webgui/lighttpd.conf
root 82362 0.0 0.4 54960 30396 - Is 13:50 0:00.08 | |-- /usr/local/bin/php-cgi
root 83021 0.0 0.4 58608 33652 - I 13:50 0:00.12 | | |-- /usr/local/bin/php-cgi
root 83050 0.0 0.4 54960 30412 - I 13:50 0:00.00 | | |-- /usr/local/bin/php-cgi
root 83117 0.0 0.4 54960 30412 - I 13:50 0:00.00 | | |-- /usr/local/bin/php-cgi
root 83262 0.0 0.4 54960 30412 - I 13:50 0:00.00 | | |-- /usr/local/bin/php-cgi
root 83357 0.0 0.4 54960 30412 - I 13:50 0:00.00 | | `-- /usr/local/bin/php-cgi
root 82549 0.0 0.4 54960 30424 - Is 13:50 0:00.08 | |-- /usr/local/bin/php-cgi
root 84675 0.0 0.5 64832 41580 - I 13:50 0:00.80 | | |-- /usr/local/bin/php-cgi
root 85321 0.0 0.5 64832 41652 - I 13:50 0:02.43 | | |-- /usr/local/bin/php-cgi
root 85551 0.0 0.5 62704 38208 - I 13:50 0:01.23 | | |-- /usr/local/bin/php-cgi
root 85821 0.0 0.4 54960 30428 - I 13:50 0:00.00 | | |-- /usr/local/bin/php-cgi
root 85989 0.0 0.4 54960 30428 - I 13:50 0:00.00 | | `-- /usr/local/bin/php-cgi
root 82807 0.0 0.4 54960 30396 - Is 13:50 0:00.08 | |-- /usr/local/bin/php-cgi
root 84184 0.0 0.4 60720 35500 - I 13:50 0:01.36 | | |-- /usr/local/bin/php-cgi
root 84450 0.0 0.4 54960 30400 - I 13:50 0:00.00 | | |-- /usr/local/bin/php-cgi
root 84621 0.0 0.4 54960 30400 - I 13:50 0:00.00 | | |-- /usr/local/bin/php-cgi
root 84829 0.0 0.4 54960 30400 - I 13:50 0:00.00 | | |-- /usr/local/bin/php-cgi
root 85030 0.0 0.4 54960 30400 - I 13:50 0:00.00 | | `-- /usr/local/bin/php-cgi
root 82960 0.0 0.4 54960 30420 - Is 13:50 0:00.08 | `-- /usr/local/bin/php-cgi
root 83590 0.0 0.6 72808 47280 - I 13:50 0:06.43 | |-- /usr/local/bin/php-cgi
root 83855 0.0 0.5 67072 43228 - I 13:50 0:02.49 | |-- /usr/local/bin/php-cgi
root 83910 0.0 0.4 54960 30424 - I 13:50 0:00.00 | |-- /usr/local/bin/php-cgi
root 84257 0.0 0.4 54960 30424 - I 13:50 0:00.00 | |-- /usr/local/bin/php-cgi
root 84586 0.0 0.4 54960 30424 - I 13:50 0:00.00 | `-- /usr/local/bin/php-cgi
see https://github.com/opnsense/core/commit/724f8494d and https://github.com/opnsense/core/commit/ec7a72f72d2 which lets lighttpd keep 4 processes with 5 children open by default it seems. This to ensure dashboard responsiveness and accommodate for long-running API locks.
Cheers,
Franco
Upgraded to 26.1-r2_2 from 25.7.11 on my home box.
Did not see too many php-cgi processes, but I did not have rapid commit enabled. System is up and running with both IPv4 and IPv6.
What I did see was two popups about errors and then this in the crash reporter:
[26-Jan-2026 18:40:11 Europe/Berlin] Error: Class "OPNsense\Mvc\Router" not found in /usr/local/opnsense/www/api.php:35
Stack trace:
There was no stack trace or panic.
I did a health check, which gave this:
***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 26.1.r2_2 (amd64) at Mon Jan 26 18:40:52 CET 2026
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 26.1.r1 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 26.1.r1 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
mimugmail (Priority: 5)
OPNsense (Priority: 11)
>>> Check installed plugins
os-acme-client 4.12
os-auto-recovery-community 1.0
os-c-icap 1.9
os-cache 1.0_1
os-caddy 2.0.4_3
os-chrony 1.5_3
os-clamav 1.8.1
os-cpu-microcode-intel 1.1
os-crowdsec 1.0.12
os-ddclient 1.29
os-dmidecode 1.2
os-dnscrypt-proxy 1.16_1
os-etpro-telemetry 1.8
os-freeradius 1.10
os-ftp-proxy 1.0_4
os-gdrive-backup 1.0
os-git-backup 1.1_2
os-haproxy 4.6_2
os-homeassistant-maxit 1.0
os-igmp-proxy 1.5_6
os-intrusion-detection-content-et-open 1.0.2_2
os-intrusion-detection-content-ptopen 1.0
os-iperf 1.0_2
os-isc-dhcp-devel 1.0_3
os-mdns-repeater 1.2
os-nextcloud-backup 1.1
os-opnarp-maxit 1.0_4
os-q-feeds-connector 1.4
os-qemu-guest-agent 1.3
os-realtek-re 1.0
os-sftp-backup 1.1_2
os-smart 2.4
os-squid 1.4
os-tayga 1.3
os-telegraf 1.12.14
os-tftp 1.0
os-theme-advanced 1.1
os-theme-cicada 1.40
os-theme-flexcolor 1.0
os-theme-rebellion 1.9.4
os-theme-solarized-community 0.4_1
os-theme-tukan 1.30
os-theme-vicuna 1.50
os-udpbroadcastrelay 1.0_6
os-upnp 1.8
os-wol 2.5_3
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 26.1.r2_2 has 67 dependencies to check.
Checking packages: .............
hostwatch-1.0.9 is not set to automatic
Checking packages: ....................................................... done
***DONE***
Don't know about the hostwatch thing. In fact, it is disabled, but last time I upgraded this way, it was enabled by default and I did not touch the setting.
Associated NAT firewall rules got disassociated and are editable, as expected.
> What I did see was two popups about errors and then this in the crash reporter:
That's the usual update hiccup. I think that at some point we will consider replacing files instead of a flat delete and install that pkg is doing, but not yet.
> Did not see too many php-cgi processes, but I did not have rapid commit enabled.
The two are not related. ;)
> Don't know about the hostwatch thing.
The missing automatic flag comes from using "pkg install" or "pkg add -f" on the command line. opnsense-revert can take care of it:
# opnsense-revert hostwatch
Cheers,
Franco
IDK if this is related, but just now, after ~16hour of running time, my system crashed and rebooted silently (no core dumps or anything and nothing in the crash reporter).
After startup, I find a host of these messages in the logs:
/usr/local/opnsense/scripts/health/updaterrd.php: The command </usr/local/bin/rrdtool create '/var/db/rrd/wireguard-traffic.rrd' --step 0 DS:'inpass:COUNTER:120:0:2500000000' DS:'outpass:COUNTER:120:0:2500000000' DS:'inblock:COUNTER:120:0:2500000000' DS:'outblock:COUNTER:120:0:2500000000' DS:'inpass6:COUNTER:120:0:2500000000' DS:'outpass6:COUNTER:120:0:2500000000' DS:'inblock6:COUNTER:120:0:2500000000' DS:'outblock6:COUNTER:120:0:2500000000' RRA:'AVERAGE:0.5:1:1200' RRA:'AVERAGE:0.5:5:720' RRA:'AVERAGE:0.5:60:1860' RRA:'AVERAGE:0.5:1440:2284'> returned exit code 1 and the output was "ERROR: step size: value must be positive"
I also have remote logging for that system. Nothing particular shows before the reboot:
2026-01-27 11_58_52-Graylog - Unsaved Search — Mozilla Firefox.png
Hello, I also have these errors;
/usr/local/opnsense/scripts/health/updaterrd.php: The command </usr/local/bin/rrdtool create '/var/db/rrd/WAN_DHCP-quality.rrd' --step 0 DS:'loss:GAUGE:120:0:2500000000' DS:'delay:GAUGE:120:0:2500000000' DS:'stddev:GAUGE:120:0:2500000000' RRA:'AVERAGE:0.5:1:1200' RRA:'AVERAGE:0.5:5:720' RRA:'AVERAGE:0.5:60:1860' RRA:'AVERAGE:0.5:1440:2284'> returned exit code 1 and the output was "ERROR: step size: value must be positive"
I think it's within netflowd, but nore sure
Reporting: Health --> Not working (No graph, but it's on)
What I have done
- reinstall rrdtool + reboot
- Firmware Audit (health) = no error
- Toggle rrd on/off/repair, etc...
OPNsense 26.1_4-amd64
Thx
Pat
This is in no way related to the topic of this thread. Better open a new one with a matching topic line.
My problem seems unrelated as of now, probably just a glitch. At least it never turned up again.
Sorry for that
Just saw a ticket was open
https://github.com/opnsense/core/issues/9686 (https://github.com/opnsense/core/issues/9686)
Since we're here can you try this commit?
# opnsense-patch https://github.com/opnsense/core/commit/6933841c6
Cheers,
Franco
Hello
# opnsense-patch 6933841
--> reboot
= Working
Thx
Great, thanks!