6rd gateway lost upon ipv4 dhcp renewal

[btusi]

I recently upgraded from 22.1 to 22.7 by installing the new version on different hardware and then restoring a configuration backup. Since then, I'm running into an issue where, when my IPv4 DHCP address renews, my IPv6 default gateway is lost (`netstat -r` shows no default for IPv6). I can temporarily fix the issue by going to the Gateway page and saving the IPv6 gateway, but it dies again once the v4 DHCP is renewed. I found a few older issues on the forum for this, but nothing for this new version, and I believe it was working fine before.

Details:
My ISP has a few oddities:
* Service is provided on a VLAN (201)
* IPv4 is provided by standard DHCP. (PPPoE is not in the picture, unlike some older deployments in my area.)
* IPv6 is provided by a 6rd tunnel.

My relevant `ifconfig` interfaces:

vlan01: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: WANQ
   options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
   ether xx:xx:xx:xx:xx:xx
   inet6 fe80::20e:c4ff:fecf:101b%vlan01 prefixlen 64 scopeid 0x9
   inet x.x.x.x netmask 0xffffff00 broadcast x.x.x.x
   groups: vlan
   vlan: 201 vlanproto: 802.1q vlanpcp: 0 parent interface: igb0
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

wan_stf: flags=4041<UP,RUNNING,LINK2> metric 0 mtu 1280
   inet6 2602:x:x:x:: prefixlen 24
   groups: stf
   v4net x.x.x.x/0 -> tv4br x.x.x.x
   nd6 options=103<PERFORMNUD,ACCEPT_RTADV,NO_DAD>


Here's a log of what happens when the DHCP renews:

<13>1 2022-08-04T22:35:37-07:00 HOSTNAME dhclient 63160 - [meta sequenceId="1"] Creating resolv.conf
<11>1 2022-08-04T22:35:37-07:00 HOSTNAME opnsense 65921 - [meta sequenceId="2"] /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'vlan01'
<11>1 2022-08-04T22:35:37-07:00 HOSTNAME opnsense 65921 - [meta sequenceId="3"] /usr/local/etc/rc.newwanip: On (IP address: x.x.x.x) (interface: WANQ[wan]) (real interface: vlan01).
<13>1 2022-08-04T22:35:37-07:00 HOSTNAME opnsense 65921 - [meta sequenceId="4"] plugins_configure hosts ()
<13>1 2022-08-04T22:35:37-07:00 HOSTNAME opnsense 65921 - [meta sequenceId="5"] plugins_configure hosts (execute task : dnsmasq_hosts_generate())
<13>1 2022-08-04T22:35:37-07:00 HOSTNAME opnsense 65921 - [meta sequenceId="6"] plugins_configure hosts (execute task : unbound_hosts_generate())
<11>1 2022-08-04T22:35:38-07:00 HOSTNAME opnsense 65921 - [meta sequenceId="7"] /usr/local/etc/rc.newwanip: rd6 lan with ipv6 address 2602:x:x:x::1 based on wan ipv4 x.x.x.x
<13>1 2022-08-04T22:35:38-07:00 HOSTNAME opnsense 65921 - [meta sequenceId="8"] plugins_configure dhcp (,inet6)
<13>1 2022-08-04T22:35:38-07:00 HOSTNAME opnsense 65921 - [meta sequenceId="9"] plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6))


And here's what happens when I save the gateway:

<13>1 2022-08-04T22:40:44-07:00 HOSTNAME configctl 59042 - [meta sequenceId="1"] event @ 1659678044.16 msg: Aug  4 22:40:44 HOSTNAME config[9866]: [2022-08-04T22:40:44-07:00][INFO] config-event: new_config /conf/backup/config-1659678044.1539.xml
<13>1 2022-08-04T22:40:44-07:00 HOSTNAME configctl 59042 - [meta sequenceId="2"] event @ 1659678044.16 exec: system event config_changed
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="3"] /system_gateways.php: ROUTING: entering configure using defaults
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="4"] /system_gateways.php: ROUTING: IPv4 default gateway set to wan
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="5"] /system_gateways.php: ROUTING: setting IPv4 default route to x.x.x.x
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="6"] /system_gateways.php: ROUTING: keeping current default gateway 'x.x.x.x'
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="7"] /system_gateways.php: ROUTING: IPv6 default gateway set to wan
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="8"] /system_gateways.php: ROUTING: setting IPv6 default route to 2602::x.x.x.x
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="9"] /system_gateways.php: ROUTING: removing /tmp/vlan01_defaultgwv6
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="10"] /system_gateways.php: ROUTING: removing /tmp/wan_stf_defaultgwv6
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="11"] /system_gateways.php: ROUTING: creating /tmp/wan_stf_defaultgwv6 using '2602::x.x.x.x'
<13>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="12"] plugins_configure monitor ()
<13>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="13"] plugins_configure monitor (execute task : dpinger_configure_do())
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="14"] /system_gateways.php: The WANQ_6RD monitor address is empty, skipping.
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="15"] /system_gateways.php: The WANQ_DHCP monitor address is empty, skipping.
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 9866 - [meta sequenceId="16"] /system_gateways.php: The WANV_4 monitor address is empty, skipping.
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 67867 - [meta sequenceId="17"] /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway 'x.x.x.x'
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 67867 - [meta sequenceId="18"] /usr/local/etc/rc.filter_configure: ROUTING: removing /tmp/wan_stf_defaultgwv6
<11>1 2022-08-04T22:40:46-07:00 HOSTNAME opnsense 67867 - [meta sequenceId="19"] /usr/local/etc/rc.filter_configure: ROUTING: creating /tmp/vlan01_defaultgwv6 using '2602::x.x.x.x'

I also have both a /tmp/vlan01_defaultgwv6 and a /tmp/wan_stf_defaultgwv6.

I admit PHP's not something I've used professionally (just some personal stuff like 20 years ago), so I'm looking into this myself but haven't found the bug quite yet. Any ideas?

[franco]

The DHCP renewal makes me suspicious so please check these two patches mentioned here first:

https://forum.opnsense.org/index.php?topic=29556.msg143071#msg143071


Cheers,
Franco

[btusi]

I applied both, and it seems to have mitigated the issue. Looking at the contents of bb4743a7322, it appears that it just made it so I don't lose the gateway every few minutes when I do a renewal, but if my ISP decides to change my IP I'll lose the gateway again. Luckily, I don't think I know of any time that's changed without me explicitly releasing/renewing, so I'll just keep my workaround in mind if it breaks again.

Thanks for the quick response!

[franco]

Thanks for the additional info.

While you are here you can break this simply by issuing

# /usr/local/etc/rc.newwanip vlan01

as is, correct?

And it starts working if you clear the cached IP first?

# rm /tmp/vlan01_oldip
# /usr/local/etc/rc.newwanip vlan01


Cheers,
Franco

[btusi]

Yes, that's correct:

root@HOSTNAME:/var/log # netstat -rn | grep default
default            x.x.x.x     UGS      vlan01
default                           2602::x:x                UGS     wan_stf
root@HOSTNAME:/var/log # /usr/local/etc/rc.newwanip vlan01
root@HOSTNAME:/var/log # netstat -rn | grep default
default            x.x.x.x     UGS      vlan01
root@HOSTNAME:/var/log # rm /tmp/vlan01_oldip
root@HOSTNAME:/var/log # /usr/local/etc/rc.newwanip vlan01
root@HOSTNAME:/var/log # netstat -rn | grep default
default            x.x.x.x     UGS      vlan01
default                           2602::x:x                UGS     wan_stf

[franco]

How about this then? https://github.com/opnsense/core/commit/e5006e9e44

# opnsense-patch e5006e9e44

Does rc.newwanip still break it or ok now?


Cheers,
Franco

[btusi]

Hey, that looks much better Thank you so much for the quick patch!

root@HOSTNAME:/var/log # opnsense-patch e5006e9e44
Fetched e5006e9e44 via https://github.com/opnsense/core
[TRUNCATED]
All patches have been applied successfully.  Have a nice day.
root@HOSTNAME:/var/log # netstat -rn | grep default
default            x.x.x.x     UGS      vlan01
default                           2602::x:x                UGS     wan_stf
root@HOSTNAME:/var/log # /usr/local/etc/rc.newwanip vlan01
root@HOSTNAME:/var/log # netstat -rn | grep default
default            x.x.x.x     UGS      vlan01
default                           2602::x:x                UGS     wan_stf

[franco]

Yay, thanks for the help. It seems that bug existed for ages, but wasn't very much triggered due to dhclient-script normally preventing spurious calls to rc.newwanip.

In any case we want to clean this up further for 23.1 to make rc.newwanip more resilient to such things.


Cheers,
Franco