Default ipv4 route drop when WAN dhcp renew

[skool]

So this brings back a working state until the next renew?

Yes, the script re-add the missing default route with the right gateway, until next renew where the gateway drops.

[franco]

Thanks, I will try to reproduce this in the lab later today.


Cheers,
Franco

[patman]

Hi!
I still have the issue with OPNsense 23.1.1_2-amd64 in strange intervals.  It always happens after several hours (probably 4 since the lease time is 28800?) of my cable modem being off. For some reason the default route is lost. Probably I know, why it is not reestablished:

Code Select Expand

<13>1 2023-02-21T06:23:10+01:00 OPNsense.lan opnsense 33755 - [meta sequenceId="6"] /usr/local/etc/rc.newwanip: No IP change detected for WAN[wan]

Code Select Expand

/usr/local/etc/rc.routing_configure fixes the issue also for me.

[franco]

Ok, so here is a temporary workaround and a debug change to see more of what is going on inside dhclient-script:

https://github.com/opnsense/core/commit/26d26e2054

# opnsense-patch 26d26e2054

Can you guys apply and see if it behaves better? I still need the log output produced by "dhclient" from the general log.


Thanks,
Franco

[patman]

Hi!

Can you guys apply and see if it behaves better? I still need the log output produced by "dhclient" from the general log.

Thanks for the patch, I have applied it this morning. It will take a few days till I can be sure that it had effect. I'll report back.

Here is the log you requested:

Code Select Expand


2023-02-22T06:24:22 Notice opnsense /usr/local/etc/rc.newwanip: No IP change detected for WAN[wan]
2023-02-22T06:24:22 Notice dhclient Creating resolv.conf
2023-02-22T06:24:22 Notice dhclient New Routers (vtnet2): 81.xxx.xx.1
2023-02-22T06:24:22 Notice dhclient New Broadcast Address (vtnet2): 81.xxx.xx.255
2023-02-22T06:24:22 Notice dhclient New Subnet Mask (vtnet2): 255.255.255.0
2023-02-22T06:24:22 Notice dhclient New IP Address (vtnet2): 81.xxx.xx.x29
2023-02-22T05:19:49 Error dhclient send_packet: No route to host
2023-02-22T04:57:12 Error dhclient send_packet: No route to host
2023-02-22T04:33:03 Error dhclient send_packet: No route to host

I do not know if relevant, but I'm running in a virtualized environment, so the WAN interface does not go down when my cable modem is off.

[franco]

Thanks, I'd love to see the debug output from the patch here to confirm, but i could also produce a similar lab condition: dhclient-script removes the main address from the interfaces and adds it back which causes the attached default route to disappear. I'm unsure when this behaviour started (FreeBSD 13 has produced more funky  problems around route setup as we've seen last year already) but perhaps we just deconstructed so much scripting that we are now at the breaking point regarding kernel and client capabilities.

https://github.com/opnsense/core/commit/90f1d1d766

# opnsense-revert opnsense && opnsense-patch 90f1d1d766


Cheers,
Franco

[chemlud]

...perhaps we just deconstructed so much scripting that we are now at the breaking point regarding kernel and client capabilities.
...
Cheers,
Franco

I've been reading this over and over again, but can't get what you are trying to say here. Heidegger? Derrida? :-D

[franco]

I'm not well-versed in philosophy ;)

The point is we have been removing race conditions, layered scripting and false assumptions from the interface code over the years to see it shrink and provide more consistent/deterministic results which ideally cause fewer bugs in the long run.

The DHCP scripting in particular lost a considerable amount of additions in 23.1 surfacing the default route bug: dhclient-script removes the same IP address and the kernel scrubs the default route because of it. That didn't matter when dhclient-script created a default route which it now cannot (22.7) in order to not break multi-WAN edge cases and on 23.1 the routing configuration was "clever" about not reloading when the IP was still the same from DHCP but missed a reload because it was defunct as per dhclient-script behaviour.


Cheers,
Franco

[patman]

Hi!

Thanks, I'd love to see the debug output from the patch here to confirm,

Ok, sorry, here is the log from this morning with patch 26d26e2054:

Code Select Expand


2023-02-23T06:24:26 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP))
2023-02-23T06:24:26 Notice opnsense /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_DHCP)
2023-02-23T06:24:26 Notice opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 81.xxx.xx.1
2023-02-23T06:24:26 Notice opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
2023-02-23T06:24:26 Notice opnsense /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
2023-02-23T06:24:26 Notice opnsense /usr/local/etc/rc.newwanip: No IP change detected for WAN[wan]
2023-02-23T06:24:26 Notice dhclient Creating resolv.conf
2023-02-23T06:24:26 Notice dhclient New Routers (vtnet2): 81.xxx.xx.1
2023-02-23T06:24:26 Notice dhclient New Broadcast Address (vtnet2): 81.xxx.xx.255
2023-02-23T06:24:26 Notice dhclient New Subnet Mask (vtnet2): 255.255.255.0
2023-02-23T06:24:26 Notice dhclient New IP Address (vtnet2): 81.xxx.xx.x29
2023-02-23T06:24:26 Notice dhclient DEBUG calling add_new_address/add_new_routes
2023-02-23T06:24:26 Notice dhclient DEBUG alias_ip_address:
2023-02-23T06:24:26 Notice dhclient DEBUG new_ip_address: 81.xxx.xx.x29
2023-02-23T06:24:26 Notice dhclient DEBUG new_ip_address: 81.xxx.xx.x29
2023-02-23T06:24:26 Notice dhclient DEBUG old_ip_address: 81.xxx.xx.x29
2023-02-23T06:24:26 Notice dhclient DEBUG entering with BOUND
2023-02-23T05:24:07 Error dhclient send_packet: No route to host
2023-02-23T05:23:27 Error dhclient send_packet: No route to host
2023-02-23T05:22:49 Error dhclient send_packet: No route to host


Default route is still here, but I'm not 100% sure as the issue did not occur everyday on my setup. Do you want me to apply patch 90f1d1d766 or should I wait still for some days to be more sure?

Thanks!

[franco]

Thanks a lot for this snippet!

I can now confirm the default route disappears when you add an existing address to the interface via ifconfig and i'm a bit baffled why that is since nothing changes except it's being treated as a new address "removing" the old and attached routes.

I think the fix still applies, feel free to try it now:

# opnsense-revert opnsense && opnsense-patch 90f1d1d766


Cheers,
Franco

[patman]

You're welcome, thanks for the help, the error was pretty annoying as it needed admin rights to fix the Internet. ;)
New patch applied, I will report back in a few days.

[franco]

Patch log is fine, thanks a lot for your help. From your previous snippet I can already see that the former workaround also addressed the missing route issue and I guess you had no problems with that patch applied?

I tested old images 21.7 and 22.1 for the default route removal behaviour on ifconfig and it seems all old versions react the same so I guess we are at the bottom of the barrel in terms of which DHCP reload behaviour is needed at which point in time.


Cheers,
Franco

[skool]

Hi,

I tested the 2 patches (with a revert between us, of course).

The first one, yesterday, didnt break (or repaired very fast) the route

Code Select Expand


<27>1 2023-02-22T13:11:20+01:00 opnsense.local dhclient 28555 - [meta sequenceId="1"] unknown dhcp option value 0x5a
<27>1 2023-02-22T13:11:20+01:00 opnsense.local dhclient 28555 - [meta sequenceId="2"] unknown dhcp option value 0x7d
<27>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 28555 - [meta sequenceId="3"] unknown dhcp option value 0x5a
<27>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 28555 - [meta sequenceId="4"] unknown dhcp option value 0x7d
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 89570 - [meta sequenceId="5"] DEBUG entering with BOUND
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 91976 - [meta sequenceId="6"] DEBUG old_ip_address:
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 94043 - [meta sequenceId="7"] DEBUG new_ip_address: 83.xx.xx.96
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 96031 - [meta sequenceId="8"] DEBUG alias_ip_address:
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 98298 - [meta sequenceId="9"] DEBUG calling add_new_address/add_new_routes
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 1724 - [meta sequenceId="10"] New IP Address (vlan0.832): 83.xx.xx.96
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 4138 - [meta sequenceId="11"] New Subnet Mask (vlan0.832): 255.255.248.0
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 7308 - [meta sequenceId="12"] New Broadcast Address (vlan0.832): 83.xx.xx.255
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 10603 - [meta sequenceId="13"] New Routers (vlan0.832): 83.xx.xx.1
<13>1 2023-02-22T13:11:22+01:00 opnsense.local dhclient 14370 - [meta sequenceId="14"] Creating resolv.conf
<13>1 2023-02-22T13:11:22+01:00 opnsense.local opnsense 31227 - [meta sequenceId="15"] /usr/local/etc/rc.newwanip: No IP change detected for WAN[wan]
<13>1 2023-02-22T13:11:22+01:00 opnsense.local opnsense 31227 - [meta sequenceId="16"] /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
<13>1 2023-02-22T13:11:22+01:00 opnsense.local opnsense 31227 - [meta sequenceId="17"] /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
<13>1 2023-02-22T13:11:22+01:00 opnsense.local opnsense 31227 - [meta sequenceId="18"] /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 83.xx.xx.1
<13>1 2023-02-22T13:11:23+01:00 opnsense.local opnsense 31227 - [meta sequenceId="19"] /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to wan
<13>1 2023-02-22T13:11:23+01:00 opnsense.local opnsense 31227 - [meta sequenceId="20"] /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
<13>1 2023-02-22T13:11:23+01:00 opnsense.local opnsense 31227 - [meta sequenceId="21"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_DHCP6)
<13>1 2023-02-22T13:11:23+01:00 opnsense.local opnsense 31227 - [meta sequenceId="22"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP6))
<13>1 2023-02-22T13:11:23+01:00 opnsense.local opnsense 31227 - [meta sequenceId="23"] /usr/local/etc/rc.newwanip: plugins_configure monitor (,WAN_DHCP)
<13>1 2023-02-22T13:11:23+01:00 opnsense.local opnsense 31227 - [meta sequenceId="24"] /usr/local/etc/rc.newwanip: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP))


But the second patch didnt worked for me, I needed to recreate my default route (with the script)

Code Select Expand


<27>1 2023-02-23T11:38:41+01:00 opnsense.local dhclient 28555 - [meta sequenceId="1"] unknown dhcp option value 0x5a
<27>1 2023-02-23T11:38:41+01:00 opnsense.local dhclient 28555 - [meta sequenceId="2"] unknown dhcp option value 0x7d
<27>1 2023-02-23T11:38:43+01:00 opnsense.local dhclient 28555 - [meta sequenceId="3"] unknown dhcp option value 0x5a
<27>1 2023-02-23T11:38:43+01:00 opnsense.local dhclient 28555 - [meta sequenceId="4"] unknown dhcp option value 0x7d
<13>1 2023-02-23T11:38:43+01:00 opnsense.local dhclient 99899 - [meta sequenceId="5"] New IP Address (vlan0.832): 83.xx.xx.96
<13>1 2023-02-23T11:38:43+01:00 opnsense.local dhclient 1486 - [meta sequenceId="6"] New Subnet Mask (vlan0.832): 255.255.248.0
<13>1 2023-02-23T11:38:43+01:00 opnsense.local dhclient 4127 - [meta sequenceId="7"] New Broadcast Address (vlan0.832): 83.xx.xx.255
<13>1 2023-02-23T11:38:43+01:00 opnsense.local dhclient 5941 - [meta sequenceId="8"] New Routers (vlan0.832): 83.xx.xx.1
<13>1 2023-02-23T11:38:43+01:00 opnsense.local dhclient 12505 - [meta sequenceId="9"] Creating resolv.conf
<13>1 2023-02-23T11:39:28+01:00 opnsense.local opnsense 50563 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2023-02-23T11:39:28+01:00 opnsense.local opnsense 50563 - [meta sequenceId="11"] /usr/local/etc/rc.routing_configure: ROUTING: IPv4 default gateway set to wan
<13>1 2023-02-23T11:39:28+01:00 opnsense.local opnsense 50563 - [meta sequenceId="12"] /usr/local/etc/rc.routing_configure: ROUTING: setting IPv4 default route to 83.xx.xx.1
<13>1 2023-02-23T11:39:28+01:00 opnsense.local opnsense 50563 - [meta sequenceId="13"] /usr/local/etc/rc.routing_configure: ROUTING: IPv6 default gateway set to wan
<13>1 2023-02-23T11:39:28+01:00 opnsense.local opnsense 50563 - [meta sequenceId="14"] /usr/local/etc/rc.routing_configure: ROUTING: skipping IPv6 default route
<13>1 2023-02-23T11:39:28+01:00 opnsense.local opnsense 50563 - [meta sequenceId="15"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (1,)
<13>1 2023-02-23T11:39:28+01:00 opnsense.local opnsense 50563 - [meta sequenceId="16"] /usr/local/etc/rc.routing_configure: plugins_configure monitor (execute task : dpinger_configure_do(1,))


I didnt had time to debug the renew today (I was on a visio meeting), but the log says that it set the gateway, but when doing a `route show default`, there was no default route.

[Xatu2]

Hi! I'm @Threefish4096 on GitHub, with this revert and patch i'm ok!

Thanks a lot for this snippet!

I can now confirm the default route disappears when you add an existing address to the interface via ifconfig and i'm a bit baffled why that is since nothing changes except it's being treated as a new address "removing" the old and attached routes.

I think the fix still applies, feel free to try it now:

# opnsense-revert opnsense && opnsense-patch 90f1d1d766


Cheers,
Franco

[waveguerilla]

Hi,

I have exactly the same problem and notice the same symptoms every day or a lot less sometimes (same Orange operator as the OP).

I'll give a try to your patch too to give you feedback.

Thank you for your help.