Duplicate DHCPv6 servers seem to be running and fighting about addresses

[cpw]

I have a static PD from my ISP (yay!) and I'm trying to hand out static IP addresses to server clients via DHCP, so I can keep them accessible, and get nice automations around DNS and stuff.

Unfortunately, the "static PD" is a tad unstable at present, and sometimes things go a bit awry, and I get a different PD from my ISP. When this happens, I notice that a duplicate DHCP server starts up, serving the new PD to my clients, while the existing one is still running!. This is, as you can imagine, causing some serious headaches. I'm not sure why this happens, but it's happened about a few dozen times since IPv6 started being available on my system - about 3 weeks ago.

Code: [Select]

[root@wall /var/log/system]# ps aux | fgrep dhcp
root     4559   0.0  0.2  31228  17460  -  Is   30Mar23      0:06.65 /usr/local/sbin/dhcp6c -c /var/etc/dhcp6c.conf -p /var/run/dhcp6c.pid -D
root    16705   0.0  0.2  25764  15032  -  Ss   09:56        0:02.06 /usr/local/bin/python3 /usr/local/opnsense/scripts/dhcp/unbound_watcher.py --domain weeksfamily.ca (python3.9)
dhcpd   46539   0.0  0.2  25712  13264  -  Ss   09:55        0:00.74 /usr/local/sbin/dhcpd -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid lagg0_vlan1 lagg0_vlan20 lagg0_vlan30 lagg0_vlan40
dhcpd   63528   0.0  0.1  22768  10328  -  Ss   Tue02        0:06.05 /usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid lagg0_vlan10
_dhcp   68325   0.0  0.0  13076   2476  -  SCs  Tue10        0:00.71 dhclient: igb2 (dhclient)
dhcpd   96030   0.0  0.1  22768  10332  -  Ss   09:55        0:00.63 /usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid lagg0_vlan10
root    96473   0.0  0.0  12756   1860  -  Is   09:55        0:00.00 daemon: /usr/local/opnsense/scripts/dhcp/prefixes.sh[97087] (daemon)
root    97087   0.0  0.0  13504   2364  -  S    09:55        0:00.05 /bin/sh /usr/local/opnsense/scripts/dhcp/prefixes.sh
root     9761   0.0  0.0  12748   1996  0  S+   12:38        0:00.00 fgrep dhcp

Here you can see that PID 63528 and 96030 are both serving v6 on the lagg0_vlan10 interface, and I get a random assortment of IPs as a result, depending which one "wins".

I suspect this is a bug, but not sure how I would go about diagnosing it, especially since it's triggered by the upstream interface getting a new PD.

[allan]

I am having the same problem under 23.1.11, and it is related to IPv6 renewals on WAN somehow. Here is my current process list.

Code: [Select]

root@gw:~ # ps aux | grep dhcpd
dhcpd   21481   0.0  0.1   25712   11752  -  Ss   Mon12       0:02.72 /usr/local/sbin/dhcpd -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid igb1 igb2_vlan2 igb2_vlan3
dhcpd   25199   0.0  0.1   22768    9956  -  Ss   Mon11       0:02.45 /usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2
dhcpd   30640   0.0  0.1   22768    9940  -  Ss   01:21       0:01.44 /usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2
dhcpd   31060   0.0  0.1   22768   10140  -  Ss   Sun01       0:05.43 /usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2 igb2_vlan3
dhcpd   63543   0.0  0.1   22768    9948  -  Ss   00:25       0:01.46 /usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2
root    87732   0.0  0.0   12748    2356  0  S+   14:04       0:00.00 grep dhcpd

I suggest checking the system log around the time dhcpd started. On mine, the earliest event today happened on 00:25 (with the second at 1:21). Here is what is in my /var/log/system/latest.log file around that time.

Code: [Select]

<13>1 2023-07-18T00:25:11-05:00 gw.net opnsense 58077 - [meta sequenceId="541"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
<13>1 2023-07-18T00:25:11-05:00 gw.net opnsense 58077 - [meta sequenceId="542"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
<11>1 2023-07-18T00:25:11-05:00 gw.net opnsense 58077 - [meta sequenceId="543"] /usr/local/etc/rc.newwanipv6: The command '/bin/kill -'TERM' '3760'' returned exit code '1', the output was 'kill: 3760: No such process'

[snip]

<11>1 2023-07-18T00:25:25-05:00 gw.net opnsense 19975 - [meta sequenceId="569"] /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.4.3-P1 Copyright 2004-2022 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Config file: /etc/dhcpdv6.conf Database file: /var/db/dhcpd6.leases PID file: /var/run/dhcpdv6.pid There's already a DHCP server running.  If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug.  These pages explain the proper process and the information we find helpful for debugging.  exiting.'
<11>1 2023-07-18T00:25:25-05:00 gw.net opnsense 19975 - [meta sequenceId="570"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 64198'
<11>1 2023-07-18T00:25:25-05:00 gw.net opnsense 38918 - [meta sequenceId="571"] /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.4.3-P1 Copyright 2004-2022 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Config file: /etc/dhcpdv6.conf Database file: /var/db/dhcpd6.leases PID file: /var/run/dhcpdv6.pid There's already a DHCP server running.  If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug.  These pages explain the proper process and the information we find helpful for debugging.  exiting.'
<11>1 2023-07-18T00:25:25-05:00 gw.net opnsense 48461 - [meta sequenceId="572"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 64198'
<11>1 2023-07-18T00:25:25-05:00 gw.net opnsense 38918 - [meta sequenceId="573"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 64198'

That log also lists the new WAN IPv6 address with the line "/usr/local/etc/rc.newwanipv6: IP renewal starting (new: xxxx, old: xxxx, interface: WAN[wan], device: igb0, force: no)". On mine, that line does not make sense since the new IPv6 address is meant for another interface and should not go to WAN. I still have to dig into that, but I want to post what I found so far.

[franco]

I've worked on such a case with a user. The conclusion was this only happens when Zenarmor is being used and it went away when disabled/removed. I couldn't find a general issue with DHCP service handling.


Cheers,
Franco

[allan]

Thanks for responding franco. I am not running Zenarmor though, only Suricata.

I found the "new:" address rc.newwanipv6 listed for WAN belonged to other interfaces - igb2_vlan2 and igb2_vlan3. These 2 VLANs are set for "Track Interface". It initially listed the IPv6 for igb2_vlan3, and switched to using the igb2_vlan2 address after I disabled IPv6 on igb2_vlan3. You can see in my ps output one dhcpd process had vlan3.

Since it continued happening with the VLAN 2 address, I changed VLAN 2 to use static IPv6 address when I posted. So now I have no interface set to "Track Interface", and I am currently monitoring logs. It has been 2 days but so far so good.

[franco]

Let me clarify... Zenarmor running which means Netmap is used which means you likely use Suricata with IPS mode enabled?


Cheers,
Franco

[allan]

Yes, I have IPS mode enabled. Do you suggest just turning off IPS, or disabling Suricata completely? I can do either.

[allan]

I turned off IPS and re-enabled Track Interface on both VLANs. The rc.newwanipv6 issue happened. I am going to completely switch off Suricata next.

Update: I notice this issue is consistently triggered by saturating my uplink (~35Mbps) with a TCP transfer (over IPv4). Completely disabling Suricata has no effect. I also tried disabling gateway monitoring since the IPv6 gateway goes offline when WAN renews its IP. None have fixed it. Switching away from Track Interface - either to Static IPv6 or disabling IPv6 altogether - has been the only solution for me so far.

[franco]

> The rc.newwanipv6 issue happened.

I thought we were talking about multiple dhcpd -6 processes? The situation you describe with rc.newwanipv6 is normal for version < 23.7. I've changed that recently, but you are likely looking at some oddity that has no impact on operation?


Cheers,
Franco

[allan]

I thought we were talking about multiple dhcpd -6 processes?

We are, but the errors on killing the process and errors in starting dhcpd were generated by rc.newwanipv6. That is what led me to think both are related.

Here are the errors from my testing yesterday. It seems there are multiple rc.newwanipv6 instances running (pid 9906, 38831, 70856), and are stepping on each other launching dhcpd processes.

Code: [Select]

<13>1 2023-07-20T16:13:30-05:00 gw.net opnsense 97682 - [meta sequenceId="30884"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,wan)
<13>1 2023-07-20T16:13:30-05:00 gw.net opnsense 97682 - [meta sequenceId="30885"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2023-07-20T16:13:30-05:00 gw.net opnsense 97682 - [meta sequenceId="30886"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())
<11>1 2023-07-20T16:13:30-05:00 gw.net opnsense 88108 - [meta sequenceId="30887"] /usr/local/etc/rc.newwanipv6: The command '/sbin/mount -r -t nullfs '/usr/local/lib/python3.9' '/var/unbound/usr/local/lib/python3.9'' returned exit code '1', the output was 'mount_nullfs: /var/unbound/usr/local/lib/python3.9: Resource deadlock avoided'
<13>1 2023-07-20T16:13:30-05:00 gw.net opnsense 9906 - [meta sequenceId="30888"] /usr/local/etc/rc.newwanipv6: IP renewal starting (new: wan_ip, old: igb2_vlan3_ip, interface: WAN[wan], device: igb0, force: no)
<13>1 2023-07-20T16:13:30-05:00 gw.net opnsense 9906 - [meta sequenceId="30889"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
<13>1 2023-07-20T16:13:30-05:00 gw.net opnsense 9906 - [meta sequenceId="30890"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
<13>1 2023-07-20T16:13:31-05:00 gw.net opnsense 52239 - [meta sequenceId="30891"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2023-07-20T16:13:32-05:00 gw.net opnsense 52239 - [meta sequenceId="30892"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,opt1))
<13>1 2023-07-20T16:13:32-05:00 gw.net opnsense 88108 - [meta sequenceId="30893"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2023-07-20T16:13:32-05:00 gw.net opnsense 88108 - [meta sequenceId="30894"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,opt1))
<11>1 2023-07-20T16:13:32-05:00 gw.net opnsense 52239 - [meta sequenceId="30895"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/updaterrd.pid' '/var/db/rrd/updaterrd.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 50943'
<13>1 2023-07-20T16:13:33-05:00 gw.net opnsense 70856 - [meta sequenceId="30896"] /usr/local/etc/rc.newwanipv6: IP renewal starting (new: wan_ip, old: igb2_vlan3_ip, interface: WAN[wan], device: igb0, force: no)
<13>1 2023-07-20T16:13:33-05:00 gw.net opnsense 70856 - [meta sequenceId="30897"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
<13>1 2023-07-20T16:13:33-05:00 gw.net opnsense 70856 - [meta sequenceId="30898"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
<11>1 2023-07-20T16:13:33-05:00 gw.net opnsense 70856 - [meta sequenceId="30899"] /usr/local/etc/rc.newwanipv6: The command '/bin/kill -'TERM' '21811'' returned exit code '1', the output was 'kill: 21811: No such process'
<13>1 2023-07-20T16:13:36-05:00 gw.net opnsense 17014 - [meta sequenceId="30900"] /usr/local/etc/rc.newwanipv6: IP renewal starting (new: wan_ip, old: igb2_vlan3_ip, interface: WAN[wan], device: igb0, force: no)
<13>1 2023-07-20T16:13:36-05:00 gw.net opnsense 17014 - [meta sequenceId="30901"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
<13>1 2023-07-20T16:13:36-05:00 gw.net opnsense 17014 - [meta sequenceId="30902"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
<11>1 2023-07-20T16:13:36-05:00 gw.net opnsense 17014 - [meta sequenceId="30903"] /usr/local/etc/rc.newwanipv6: The command '/bin/kill -'TERM' '21811'' returned exit code '1', the output was 'kill: 21811: No such process'
<13>1 2023-07-20T16:13:39-05:00 gw.net opnsense 38831 - [meta sequenceId="30904"] /usr/local/etc/rc.newwanipv6: IP renewal starting (new: wan_ip, old: igb2_vlan3_ip, interface: WAN[wan], device: igb0, force: no)
<13>1 2023-07-20T16:13:39-05:00 gw.net opnsense 38831 - [meta sequenceId="30905"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
<13>1 2023-07-20T16:13:39-05:00 gw.net opnsense 38831 - [meta sequenceId="30906"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
<11>1 2023-07-20T16:13:39-05:00 gw.net opnsense 38831 - [meta sequenceId="30907"] /usr/local/etc/rc.newwanipv6: The command '/bin/kill -'TERM' '21811'' returned exit code '1', the output was 'kill: 21811: No such process'
<13>1 2023-07-20T16:13:40-05:00 gw.net opnsense 97682 - [meta sequenceId="30908"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : opendns_configure_do())
<13>1 2023-07-20T16:13:40-05:00 gw.net opnsense 97682 - [meta sequenceId="30909"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : openssh_configure_do(,wan))
<13>1 2023-07-20T16:13:40-05:00 gw.net opnsense 97682 - [meta sequenceId="30910"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : unbound_configure_do(,wan))
<13>1 2023-07-20T16:13:42-05:00 gw.net opnsense 86168 - [meta sequenceId="30911"] /usr/local/etc/rc.newwanipv6: IP renewal starting (new: wan_ip, old: igb2_vlan3_ip, interface: WAN[wan], device: igb0, force: no)
<13>1 2023-07-20T16:13:42-05:00 gw.net opnsense 86168 - [meta sequenceId="30912"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
<13>1 2023-07-20T16:13:42-05:00 gw.net opnsense 86168 - [meta sequenceId="30913"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
<11>1 2023-07-20T16:13:42-05:00 gw.net opnsense 86168 - [meta sequenceId="30914"] /usr/local/etc/rc.newwanipv6: The command '/bin/kill -'TERM' '21811'' returned exit code '1', the output was 'kill: 21811: No such process'
<13>1 2023-07-20T16:13:43-05:00 gw.net opnsense 97682 - [meta sequenceId="30915"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : vxlan_configure_do())
<13>1 2023-07-20T16:13:43-05:00 gw.net opnsense 97682 - [meta sequenceId="30916"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : webgui_configure_do(,wan))
<13>1 2023-07-20T16:13:43-05:00 gw.net opnsense 97682 - [meta sequenceId="30917"] /usr/local/etc/rc.newwanipv6: plugins_configure vpn (,opt2)
<13>1 2023-07-20T16:13:43-05:00 gw.net opnsense 97682 - [meta sequenceId="30918"] /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : ipsec_configure_do(,opt2))
<13>1 2023-07-20T16:13:43-05:00 gw.net opnsense 97682 - [meta sequenceId="30919"] /usr/local/etc/rc.newwanipv6: plugins_configure vpn (execute task : openvpn_configure_do(,opt2))
<13>1 2023-07-20T16:13:43-05:00 gw.net opnsense 97682 - [meta sequenceId="30920"] /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface Guest.
<13>1 2023-07-20T16:13:43-05:00 gw.net opnsense 97682 - [meta sequenceId="30921"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (,opt2)
<13>1 2023-07-20T16:13:43-05:00 gw.net opnsense 97682 - [meta sequenceId="30922"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : dnsmasq_configure_do())
<13>1 2023-07-20T16:13:43-05:00 gw.net opnsense 97682 - [meta sequenceId="30923"] /usr/local/etc/rc.newwanipv6: plugins_configure newwanip (execute task : ntpd_configure_do())
<13>1 2023-07-20T16:13:45-05:00 gw.net opnsense 34197 - [meta sequenceId="30924"] /usr/local/etc/rc.newwanipv6: IP renewal starting (new: wan_ip, old: igb2_vlan3_ip, interface: WAN[wan], device: igb0, force: no)
<13>1 2023-07-20T16:13:45-05:00 gw.net opnsense 34197 - [meta sequenceId="30925"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (,inet6)
<13>1 2023-07-20T16:13:45-05:00 gw.net opnsense 34197 - [meta sequenceId="30926"] /usr/local/etc/rc.newwanipv6: plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))
<11>1 2023-07-20T16:13:45-05:00 gw.net opnsense 34197 - [meta sequenceId="30927"] /usr/local/etc/rc.newwanipv6: The command '/bin/kill -'TERM' '21811'' returned exit code '1', the output was 'kill: 21811: No such process'
<11>1 2023-07-20T16:13:47-05:00 gw.net opnsense 9906 - [meta sequenceId="30928"] /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2 igb2_vlan3' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.4.3-P1 Copyright 2004-2022 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Config file: /etc/dhcpdv6.conf Database file: /var/db/dhcpd6.leases PID file: /var/run/dhcpdv6.pid There's already a DHCP server running.  If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug.  These pages explain the proper process and the information we find helpful for debugging.  exiting.'
<11>1 2023-07-20T16:13:47-05:00 gw.net opnsense 9906 - [meta sequenceId="30929"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 81552'
<11>1 2023-07-20T16:13:48-05:00 gw.net opnsense 86168 - [meta sequenceId="30930"] /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2 igb2_vlan3' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.4.3-P1 Copyright 2004-2022 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Config file: /etc/dhcpdv6.conf Database file: /var/db/dhcpd6.leases PID file: /var/run/dhcpdv6.pid There's already a DHCP server running.  If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug.  These pages explain the proper process and the information we find helpful for debugging.  exiting.'
<11>1 2023-07-20T16:13:48-05:00 gw.net opnsense 86168 - [meta sequenceId="30931"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 81552'
<11>1 2023-07-20T16:13:48-05:00 gw.net opnsense 70856 - [meta sequenceId="30932"] /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2 igb2_vlan3' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.4.3-P1 Copyright 2004-2022 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Config file: /etc/dhcpdv6.conf Database file: /var/db/dhcpd6.leases PID file: /var/run/dhcpdv6.pid There's already a DHCP server running.  If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug.  These pages explain the proper process and the information we find helpful for debugging.  exiting.'
<11>1 2023-07-20T16:13:48-05:00 gw.net opnsense 38831 - [meta sequenceId="30933"] /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2 igb2_vlan3' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.4.3-P1 Copyright 2004-2022 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Config file: /etc/dhcpdv6.conf Database file: /var/db/dhcpd6.leases PID file: /var/run/dhcpdv6.pid There's already a DHCP server running.  If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug.  These pages explain the proper process and the information we find helpful for debugging.  exiting.'
<11>1 2023-07-20T16:13:48-05:00 gw.net opnsense 70856 - [meta sequenceId="30934"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 81552'
<11>1 2023-07-20T16:13:48-05:00 gw.net opnsense 38831 - [meta sequenceId="30935"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 81552'
<11>1 2023-07-20T16:13:48-05:00 gw.net opnsense 34197 - [meta sequenceId="30936"] /usr/local/etc/rc.newwanipv6: The command '/usr/local/sbin/dhcpd -6 -user dhcpd -group dhcpd -chroot /var/dhcpd -cf /etc/dhcpdv6.conf -pf /var/run/dhcpdv6.pid igb1 igb2_vlan2 igb2_vlan3' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.4.3-P1 Copyright 2004-2022 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Config file: /etc/dhcpdv6.conf Database file: /var/db/dhcpd6.leases PID file: /var/run/dhcpdv6.pid There's already a DHCP server running.  If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug.  These pages explain the proper process and the information we find helpful for debugging.  exiting.'
<11>1 2023-07-20T16:13:48-05:00 gw.net opnsense 34197 - [meta sequenceId="30937"] /usr/local/etc/rc.newwanipv6: The command '/usr/sbin/daemon -f -p '/var/run/dhcpleases6.pid' '/usr/local/opnsense/scripts/dhcp/prefixes.sh'' returned exit code '3', the output was 'daemon: process already running, pid: 81552'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 17014 - [meta sequenceId="30938"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 86168 - [meta sequenceId="30939"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 38831 - [meta sequenceId="30940"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 9906 - [meta sequenceId="30941"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 34197 - [meta sequenceId="30942"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 17014 - [meta sequenceId="30943"] /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 17014 - [meta sequenceId="30944"] /usr/local/etc/rc.newwanipv6: ROUTING: keeping current inet6 default gateway 'fe80::fad2:acff:febf:695b%igb0'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 17014 - [meta sequenceId="30945"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 17014 - [meta sequenceId="30946"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 9906 - [meta sequenceId="30947"] /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 9906 - [meta sequenceId="30948"] /usr/local/etc/rc.newwanipv6: ROUTING: keeping current inet6 default gateway 'fe80::fad2:acff:febf:695b%igb0'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 9906 - [meta sequenceId="30949"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 9906 - [meta sequenceId="30950"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 86168 - [meta sequenceId="30951"] /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 86168 - [meta sequenceId="30952"] /usr/local/etc/rc.newwanipv6: ROUTING: keeping current inet6 default gateway 'fe80::fad2:acff:febf:695b%igb0'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 86168 - [meta sequenceId="30953"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 86168 - [meta sequenceId="30954"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 70856 - [meta sequenceId="30955"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'wan'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 17014 - [meta sequenceId="30956"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'opt2'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 38831 - [meta sequenceId="30957"] /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 34197 - [meta sequenceId="30958"] /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 38831 - [meta sequenceId="30959"] /usr/local/etc/rc.newwanipv6: ROUTING: keeping current inet6 default gateway 'fe80::fad2:acff:febf:695b%igb0'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 38831 - [meta sequenceId="30960"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 38831 - [meta sequenceId="30961"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 34197 - [meta sequenceId="30962"] /usr/local/etc/rc.newwanipv6: ROUTING: keeping current inet6 default gateway 'fe80::fad2:acff:febf:695b%igb0'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 34197 - [meta sequenceId="30963"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 34197 - [meta sequenceId="30964"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 70856 - [meta sequenceId="30965"] /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 70856 - [meta sequenceId="30966"] /usr/local/etc/rc.newwanipv6: ROUTING: keeping current inet6 default gateway 'fe80::fad2:acff:febf:695b%igb0'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 70856 - [meta sequenceId="30967"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (,WAN_DHCP6)
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 70856 - [meta sequenceId="30968"] /usr/local/etc/rc.newwanipv6: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 86168 - [meta sequenceId="30969"] /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'opt2'
<13>1 2023-07-20T16:13:49-05:00 gw.net opnsense 17014 - [meta sequenceId="30970"] /usr/local/etc/rc.newwanipv6: ROUTING: configuring inet6 default gateway on wan