RC1: hundreds of rc.newwanipv6 processes bringing system down

[Patrick M. Hausen]

Hi all,

after upgrade from 25.7.11_2 to 26.1r1 everything looked good at first. I did not yet try the rule migration but intended to wait for RC2 with all the fixes in that specific area.

Half an hour later Internet was down. SSH to the box still working, system quite sluggish, dashboard widgets failing to load.

A couple of hundred processes like this:

Code Select Expand

/usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe0 force

"killall -9 php" made the system responsive again for a short while but the processes kept piling up.

Anything specific in the log I should look for?

With 25.7 running, this is the dhcp6d.conf:

Code Select Expand

interface pppoe0 {
  send ia-na 2; # request stateful address
  send ia-pd 2; # request prefix delegation
  request domain-name-servers;
  request domain-name;
  script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
id-assoc na 2 { };
id-assoc pd 2 {
  prefix ::/56 infinity;
};

I'm a bit puzzled by that "request domain-name-servers;" - is that hard coded? I could not find a way to disable it, anywhere and I certainly do not want any DNS servers, be it v4 or v6 from my ISP.

I isolated the logs for a single PID when the system was running 26.1r1:

Code Select Expand

root@opnsense:/var/log/system # grep 56100 *
latest.log:<29>1 2026-01-25T14:53:57+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="701"] Sending Solicit
latest.log:<27>1 2026-01-25T14:53:57+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="702"] transmit failed: Can't assign requested address
latest.log:<29>1 2026-01-25T14:53:58+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="705"] Sending Solicit
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="721"] Sending Request
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="722"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="723"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="724"] failed to update an address ::
latest.log:<29>1 2026-01-25T14:54:00+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="732"] Sending Solicit
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="733"] Sending Request
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="734"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="735"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="736"] failed to update an address ::
latest.log:<29>1 2026-01-25T14:54:02+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="741"] Sending Solicit
latest.log:<29>1 2026-01-25T14:54:03+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="744"] Sending Request
latest.log:<29>1 2026-01-25T14:54:06+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="746"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:54:06+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="747"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:54:06+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="748"] failed to update an address ::
latest.log:<29>1 2026-01-25T14:54:07+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="757"] Sending Solicit
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="759"] Sending Request
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="760"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="761"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="762"] failed to update an address ::
[...]
system_20260125.log:<29>1 2026-01-25T16:12:17+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20634"] Sending Solicit
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20640"] Sending Request
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20641"] Received REPLY for REQUEST
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20642"] failed to remove an address on pppoe0: Can't assign requested address
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20643"] failed to update an address ::
system_20260125.log:<29>1 2026-01-25T16:12:19+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20646"] Sending Solicit
root@opnsense:/var/log/system #

Kind regards,
Patrick

[nero355]

Half an hour later Internet was down. SSH to the box still working, system quite sluggish, dashboard widgets failing to load.

A couple of hundred processes like this:

Code Select Expand

/usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe0 force

"killall -9 php" made the system responsive again for a short while but the processes kept piling up.

Sounds similar to : https://forum.opnsense.org/index.php?msg=257256 ??

[Patrick M. Hausen]

I don't use Suricata or Zenarmor.

[franco]

https://forum.opnsense.org/index.php?topic=50500.0 appears to be the same and I've heard that privately before once too.

The questions are these:

1. Did you configure anything else than what 25.7.x already offered?

2. Do you have the full system log around the incident to inspect?

It looks like a loop we've seen many years ago in PPPoE vs. rc.newwanipv6 script and there haven't been much changes.  I'm taking an educated guess with a commit in a second we can easily flush to 26.1-RC2.


Cheers,
Franco

[Patrick M. Hausen]

I uninstalled the ISC DHCP plugin, but did not change anything else.

I'll send you an email.

Thanks!
Patrick

[franco]

Thanks, that would further fuel the suspicion.  I'll want to add https://github.com/opnsense/core/commit/3248b4d2315f to RC2 and see how it goes (with extra logging).

Going through your log in a second as well and reporting back.


Cheers,
Franco

[Patrick M. Hausen]

Situation unchanged with RC2. Already sent a debug log to Franco.

[franco]

# opnsense-patch https://github.com/opnsense/core/commit/9a80c6ddb29

This should make it behave as before on 25.7.x, best done with a reboot right after apply.

If that's the case we found a very old bug in dhcp6c.


Cheers,
Franco

[Patrick M. Hausen]

I already had a complete post telling you "nope", but then decided to give it one more try, because with your hint in the private email I had enabled rapid commit.

So the results so far:

RC2 with the patch, same configuration as with 25.7: fine
RC2 with the patch, rapid commit enabled: dpcp6c processes spawning again

I'll leave the patched RC2 running for now.

HTH,
Patrick

P.S. Why is the just booted system with no UI session active running 25 php-cgi processes?

[franco]

Rapid-commit (new in 26.1) may run into the same issue if the server refused the request altogether. It's probably going to remain optional anyway.  Let he hotfix that one then.

About php-cgi... this is normal

Code Select Expand

root    82341   0.0  0.1  23728  11060  -  S    13:50      0:01.69 |-- /usr/local/sbin/lighttpd -f /usr/local/etc/lighttpd_webgui/lighttpd.conf
root    82362   0.0  0.4  54960  30396  -  Is   13:50      0:00.08 | |-- /usr/local/bin/php-cgi
root    83021   0.0  0.4  58608  33652  -  I    13:50      0:00.12 | | |-- /usr/local/bin/php-cgi
root    83050   0.0  0.4  54960  30412  -  I    13:50      0:00.00 | | |-- /usr/local/bin/php-cgi
root    83117   0.0  0.4  54960  30412  -  I    13:50      0:00.00 | | |-- /usr/local/bin/php-cgi
root    83262   0.0  0.4  54960  30412  -  I    13:50      0:00.00 | | |-- /usr/local/bin/php-cgi
root    83357   0.0  0.4  54960  30412  -  I    13:50      0:00.00 | | `-- /usr/local/bin/php-cgi
root    82549   0.0  0.4  54960  30424  -  Is   13:50      0:00.08 | |-- /usr/local/bin/php-cgi
root    84675   0.0  0.5  64832  41580  -  I    13:50      0:00.80 | | |-- /usr/local/bin/php-cgi
root    85321   0.0  0.5  64832  41652  -  I    13:50      0:02.43 | | |-- /usr/local/bin/php-cgi
root    85551   0.0  0.5  62704  38208  -  I    13:50      0:01.23 | | |-- /usr/local/bin/php-cgi
root    85821   0.0  0.4  54960  30428  -  I    13:50      0:00.00 | | |-- /usr/local/bin/php-cgi
root    85989   0.0  0.4  54960  30428  -  I    13:50      0:00.00 | | `-- /usr/local/bin/php-cgi
root    82807   0.0  0.4  54960  30396  -  Is   13:50      0:00.08 | |-- /usr/local/bin/php-cgi
root    84184   0.0  0.4  60720  35500  -  I    13:50      0:01.36 | | |-- /usr/local/bin/php-cgi
root    84450   0.0  0.4  54960  30400  -  I    13:50      0:00.00 | | |-- /usr/local/bin/php-cgi
root    84621   0.0  0.4  54960  30400  -  I    13:50      0:00.00 | | |-- /usr/local/bin/php-cgi
root    84829   0.0  0.4  54960  30400  -  I    13:50      0:00.00 | | |-- /usr/local/bin/php-cgi
root    85030   0.0  0.4  54960  30400  -  I    13:50      0:00.00 | | `-- /usr/local/bin/php-cgi
root    82960   0.0  0.4  54960  30420  -  Is   13:50      0:00.08 | `-- /usr/local/bin/php-cgi
root    83590   0.0  0.6  72808  47280  -  I    13:50      0:06.43 |   |-- /usr/local/bin/php-cgi
root    83855   0.0  0.5  67072  43228  -  I    13:50      0:02.49 |   |-- /usr/local/bin/php-cgi
root    83910   0.0  0.4  54960  30424  -  I    13:50      0:00.00 |   |-- /usr/local/bin/php-cgi
root    84257   0.0  0.4  54960  30424  -  I    13:50      0:00.00 |   |-- /usr/local/bin/php-cgi
root    84586   0.0  0.4  54960  30424  -  I    13:50      0:00.00 |   `-- /usr/local/bin/php-cgi
see https://github.com/opnsense/core/commit/724f8494d and https://github.com/opnsense/core/commit/ec7a72f72d2 which lets lighttpd keep 4 processes with 5 children open by default it seems. This to ensure dashboard responsiveness and accommodate for long-running API locks.


Cheers,
Franco

[meyergru]

Upgraded to 26.1-r2_2 from 25.7.11 on my home box.

Did not see too many php-cgi processes, but I did not have rapid commit enabled. System is up and running with both IPv4 and IPv6.



What I did see was two popups about errors and then this in the crash reporter:

Code Select Expand

[26-Jan-2026 18:40:11 Europe/Berlin] Error: Class "OPNsense\Mvc\Router" not found in /usr/local/opnsense/www/api.php:35
Stack trace:


There was no stack trace or panic.

I did a health check, which gave this:

Code Select Expand

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 26.1.r2_2 (amd64) at Mon Jan 26 18:40:52 CET 2026
>>> Root file system: zroot/ROOT/default
>>> Check installed kernel version
Version 26.1.r1 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 26.1.r1 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
mimugmail (Priority: 5)
OPNsense (Priority: 11)
>>> Check installed plugins
os-acme-client 4.12
os-auto-recovery-community 1.0
os-c-icap 1.9
os-cache 1.0_1
os-caddy 2.0.4_3
os-chrony 1.5_3
os-clamav 1.8.1
os-cpu-microcode-intel 1.1
os-crowdsec 1.0.12
os-ddclient 1.29
os-dmidecode 1.2
os-dnscrypt-proxy 1.16_1
os-etpro-telemetry 1.8
os-freeradius 1.10
os-ftp-proxy 1.0_4
os-gdrive-backup 1.0
os-git-backup 1.1_2
os-haproxy 4.6_2
os-homeassistant-maxit 1.0
os-igmp-proxy 1.5_6
os-intrusion-detection-content-et-open 1.0.2_2
os-intrusion-detection-content-ptopen 1.0
os-iperf 1.0_2
os-isc-dhcp-devel 1.0_3
os-mdns-repeater 1.2
os-nextcloud-backup 1.1
os-opnarp-maxit 1.0_4
os-q-feeds-connector 1.4
os-qemu-guest-agent 1.3
os-realtek-re 1.0
os-sftp-backup 1.1_2
os-smart 2.4
os-squid 1.4
os-tayga 1.3
os-telegraf 1.12.14
os-tftp 1.0
os-theme-advanced 1.1
os-theme-cicada 1.40
os-theme-flexcolor 1.0
os-theme-rebellion 1.9.4
os-theme-solarized-community 0.4_1
os-theme-tukan 1.30
os-theme-vicuna 1.50
os-udpbroadcastrelay 1.0_6
os-upnp 1.8
os-wol 2.5_3
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" at 26.1.r2_2 has 67 dependencies to check.
Checking packages: .............
hostwatch-1.0.9 is not set to automatic
Checking packages: ....................................................... done
***DONE***
Don't know about the hostwatch thing. In fact, it is disabled, but last time I upgraded this way, it was enabled by default and I did not touch the setting.

Associated NAT firewall rules got disassociated and are editable, as expected.