RC1: hundreds of rc.newwanipv6 processes bringing system down

[Patrick M. Hausen]

Hi all,

after upgrade from 25.7.11_2 to 26.1r1 everything looked good at first. I did not yet try the rule migration but intended to wait for RC2 with all the fixes in that specific area.

Half an hour later Internet was down. SSH to the box still working, system quite sluggish, dashboard widgets failing to load.

A couple of hundred processes like this:

Code Select Expand

/usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe0 force

"killall -9 php" made the system responsive again for a short while but the processes kept piling up.

Anything specific in the log I should look for?

With 25.7 running, this is the dhcp6d.conf:

Code Select Expand

interface pppoe0 {
  send ia-na 2; # request stateful address
  send ia-pd 2; # request prefix delegation
  request domain-name-servers;
  request domain-name;
  script "/var/etc/dhcp6c_wan_script.sh"; # we'd like some nameservers please
};
id-assoc na 2 { };
id-assoc pd 2 {
  prefix ::/56 infinity;
};

I'm a bit puzzled by that "request domain-name-servers;" - is that hard coded? I could not find a way to disable it, anywhere and I certainly do not want any DNS servers, be it v4 or v6 from my ISP.

I isolated the logs for a single PID when the system was running 26.1r1:

Code Select Expand

root@opnsense:/var/log/system # grep 56100 *
latest.log:<29>1 2026-01-25T14:53:57+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="701"] Sending Solicit
latest.log:<27>1 2026-01-25T14:53:57+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="702"] transmit failed: Can't assign requested address
latest.log:<29>1 2026-01-25T14:53:58+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="705"] Sending Solicit
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="721"] Sending Request
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="722"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="723"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:53:59+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="724"] failed to update an address ::
latest.log:<29>1 2026-01-25T14:54:00+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="732"] Sending Solicit
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="733"] Sending Request
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="734"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="735"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:54:01+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="736"] failed to update an address ::
latest.log:<29>1 2026-01-25T14:54:02+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="741"] Sending Solicit
latest.log:<29>1 2026-01-25T14:54:03+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="744"] Sending Request
latest.log:<29>1 2026-01-25T14:54:06+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="746"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:54:06+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="747"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:54:06+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="748"] failed to update an address ::
latest.log:<29>1 2026-01-25T14:54:07+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="757"] Sending Solicit
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="759"] Sending Request
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="760"] Received REPLY for REQUEST
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="761"] failed to remove an address on pppoe0: Can't assign requested address
latest.log:<29>1 2026-01-25T14:54:08+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="762"] failed to update an address ::
[...]
system_20260125.log:<29>1 2026-01-25T16:12:17+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20634"] Sending Solicit
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20640"] Sending Request
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20641"] Received REPLY for REQUEST
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20642"] failed to remove an address on pppoe0: Can't assign requested address
system_20260125.log:<29>1 2026-01-25T16:12:18+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20643"] failed to update an address ::
system_20260125.log:<29>1 2026-01-25T16:12:19+01:00 opnsense.ettlingen.hausen.com dhcp6c 56100 - [meta sequenceId="20646"] Sending Solicit
root@opnsense:/var/log/system #

Kind regards,
Patrick

[nero355]

Half an hour later Internet was down. SSH to the box still working, system quite sluggish, dashboard widgets failing to load.

A couple of hundred processes like this:

Code Select Expand

/usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe0 force

"killall -9 php" made the system responsive again for a short while but the processes kept piling up.

Sounds similar to : https://forum.opnsense.org/index.php?msg=257256 ??

[Patrick M. Hausen]

I don't use Suricata or Zenarmor.

[franco]

https://forum.opnsense.org/index.php?topic=50500.0 appears to be the same and I've heard that privately before once too.

The questions are these:

1. Did you configure anything else than what 25.7.x already offered?

2. Do you have the full system log around the incident to inspect?

It looks like a loop we've seen many years ago in PPPoE vs. rc.newwanipv6 script and there haven't been much changes.  I'm taking an educated guess with a commit in a second we can easily flush to 26.1-RC2.


Cheers,
Franco

[Patrick M. Hausen]

I uninstalled the ISC DHCP plugin, but did not change anything else.

I'll send you an email.

Thanks!
Patrick

[franco]

Thanks, that would further fuel the suspicion.  I'll want to add https://github.com/opnsense/core/commit/3248b4d2315f to RC2 and see how it goes (with extra logging).

Going through your log in a second as well and reporting back.


Cheers,
Franco

[Patrick M. Hausen]

Situation unchanged with RC2. Already sent a debug log to Franco.

[franco]

# opnsense-patch https://github.com/opnsense/core/commit/9a80c6ddb29

This should make it behave as before on 25.7.x, best done with a reboot right after apply.

If that's the case we found a very old bug in dhcp6c.


Cheers,
Franco

[Patrick M. Hausen]

I already had a complete post telling you "nope", but then decided to give it one more try, because with your hint in the private email I had enabled rapid commit.

So the results so far:

RC2 with the patch, same configuration as with 25.7: fine
RC2 with the patch, rapid commit enabled: dpcp6c processes spawning again

I'll leave the patched RC2 running for now.

HTH,
Patrick

P.S. Why is the just booted system with no UI session active running 25 php-cgi processes?