Default ipv4 route drop when WAN dhcp renew

[skool]

Hi all,
Since 23.1 update, I have an error every day. Default IPv4 route drop every 24h when my WAN interface renew this DHCP lease.
I'm using this config : https://docs.opnsense.org/manual/how-tos/orange_fr_fttp.html for Orange FTTH provider (in France).
I tried to reinstall and reimport my config, and recreating the VLAN interface from scratch, but without any difference.

I see these logs related to dhcp.

Code: [Select]

<27>1 2023-02-06T11:04:09+01:00 opnsense.xxx dhclient 61916 - [meta sequenceId="1"] unknown dhcp option value 0x5a
<27>1 2023-02-06T11:04:09+01:00 opnsense.xxx dhclient 61916 - [meta sequenceId="2"] unknown dhcp option value 0x7d
<27>1 2023-02-06T11:04:11+01:00 opnsense.xxx dhclient 61916 - [meta sequenceId="3"] unknown dhcp option value 0x5a
<27>1 2023-02-06T11:04:11+01:00 opnsense.xxx dhclient 61916 - [meta sequenceId="4"] unknown dhcp option value 0x7d
<13>1 2023-02-06T11:04:11+01:00 opnsense.xxx dhclient 37422 - [meta sequenceId="5"] New IP Address (vlan0.832): 83.202.xx.xx
<13>1 2023-02-06T11:04:11+01:00 opnsense.xxx dhclient 39678 - [meta sequenceId="6"] New Subnet Mask (vlan0.832): 255.255.248.0
<13>1 2023-02-06T11:04:11+01:00 opnsense.xxx dhclient 42657 - [meta sequenceId="7"] New Broadcast Address (vlan0.832): 83.202.xx.255
<13>1 2023-02-06T11:04:11+01:00 opnsense.xxx dhclient 44227 - [meta sequenceId="8"] New Routers (vlan0.832): 83.202.xx.1
<13>1 2023-02-06T11:04:11+01:00 opnsense.xxx dhclient 49809 - [meta sequenceId="9"] Creating resolv.conf
<13>1 2023-02-06T11:04:11+01:00 opnsense.xxx opnsense 66635 - [meta sequenceId="10"] /usr/local/etc/rc.newwanip: No IP change detected for WAN[wan]

DHCP lease got from my ISP is for 24h, but when renewing, it seems that the routes related to this link is drop off.
I need, to fix it, to do one of this step :
- manually add default route
- click to renew dhcp lease
- unplug/replug the network cable

IPv6 have no issue, but the DHCP6 lease seems to be longer than the IPv4 one.

I dont find more detailed logs…
Is someone have the same behaviour ? I didnt seen an issue on github.
Thanks

[skool]

I see 2 files used when dhcp renew an IP

https://github.com/opnsense/core/blob/master/src/etc/rc.newwanip
https://github.com/opnsense/core/blob/master/src/opnsense/scripts/interfaces/dhclient-script
Both have multiple changes between 22.7 and 23.1, regarding the way they manage ips, routes, etc…
I dont understand why my route is removed during the process.
I suppose that a few instants, interface is down, so the system remove the route, and as the new IP is the same than the old, the 2 scripts didnt recreate the route.
But I cant confirm that, and why it seems I'm the only one to have this issue ?

The setup seems classic for a public FAI, a dhclient on interface with vlan (and some custom options but probably not related).

Is there a way to track all route changes ?

[cayenne]

Hello,

I have the same problem losing the default route but only in IPv6.
In IPv4, the WAN interface is configured as static IP.

https://forum.opnsense.org/index.php?topic=32263

[patman]

Hmm, that sounds similar to my issue https://forum.opnsense.org/index.php?topic=32439.0. Is there a way to trigger the issue?

[debacler]

This is also similar to an issue I am having after upgrading to 23.1. My WAN has no special configuration, just DHCP for IPv4 and IPv6.

Approx. every 24 hours the default IPv4 route vanishes. A reboot of opnsense or disabling and reenabling the IPv4 gateway brings it back and restores connectivity. Haven't found anything useful in the logs yet.

I have found several similar complaints from other users about this behavior after the 23.1 upgrade, but no resolutions.

[franco]

Hmm, so I wonder who is dropping the default route here? dhclient-script was changed to not touch it a while ago. Could it be that no actual route is being pushed by the DHCP server and it is only sending static routes?

Could someone post their # cat /var/db/dhclient.leases.* file?


Thanks,
Franco

[franco]

PS: we need a little more logging prior to the renew and the infamous "No IP change detected for WAN[wan]"

[debacler]

Here's the contents, thanks:

Code: [Select]

root@OPNsense:~ # cat /var/db/dhclient.leases.igb0
lease {
  interface "igb0";
  fixed-address 209.xxx.xxx.218;
  option subnet-mask 255.255.255.0;
  option routers 209.xxx.xxx.1;
  option domain-name-servers 209.206.64.138,209.206.64.246;
  option host-name "xxxxxxxx";
  option domain-name "elevatefiber.com";
  option dhcp-lease-time 7200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 172.20.99.46;
  renew 4 2023/2/16 13:25:05;
  rebind 4 2023/2/16 14:10:05;
  expire 4 2023/2/16 14:25:05;
}
lease {
  interface "igb0";
  fixed-address 209.xxx.xxx.218;
  option subnet-mask 255.255.255.0;
  option routers 209.xxx.xxx.1;
  option domain-name-servers 209.206.64.138,209.206.64.246;
  option host-name "xxxxxxxx";
  option domain-name "elevatefiber.com";
  option dhcp-lease-time 7200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 172.20.99.46;
  renew 4 2023/2/16 14:17:00;
  rebind 4 2023/2/16 15:02:00;
  expire 4 2023/2/16 15:17:00;
}
lease {
  interface "igb0";
  fixed-address 209.xxx.xxx.218;
  option subnet-mask 255.255.255.0;
  option routers 209.xxx.xxx.1;
  option domain-name-servers 209.206.64.138,209.206.64.246;
  option host-name "xxxxxxxx";
  option domain-name "elevatefiber.com";
  option dhcp-lease-time 7200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 172.20.99.46;
  renew 4 2023/2/16 16:07:55;
  rebind 4 2023/2/16 16:52:55;
  expire 4 2023/2/16 17:07:55;
}
lease {
  interface "igb0";
  fixed-address 209.xxx.xxx.218;
  option subnet-mask 255.255.255.0;
  option routers 209.xxx.xxx.1;
  option domain-name-servers 209.206.64.138,209.206.64.246;
  option host-name "xxxxxxxx";
  option domain-name "elevatefiber.com";
  option dhcp-lease-time 7200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 172.20.99.46;
  renew 4 2023/2/16 18:01:20;
  rebind 4 2023/2/16 18:46:20;
  expire 4 2023/2/16 19:01:20;
}


[axsdenied]

This is also similar to an issue I am having after upgrading to 23.1. My WAN has no special configuration, just DHCP for IPv4 and IPv6.

Approx. every 24 hours the default IPv4 route vanishes. A reboot of opnsense or disabling and reenabling the IPv4 gateway brings it back and restores connectivity. Haven't found anything useful in the logs yet.

I have found several similar complaints from other users about this behavior after the 23.1 upgrade, but no resolutions.

I wonder if the root cause of this issue related to an issue I described: https://forum.opnsense.org/index.php?topic=32490.0

[skool]

Hi,
That's my dhcp lease db

Code: [Select]

lease {
  interface "vlan0.832";
  fixed-address 83.xx.xx.96;
  next-server 80.10.247.48;
  option subnet-mask 255.255.248.0;
  option routers 83.xx.xx.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option domain-name "orange.fr";
  option broadcast-address 83.xx.xx.255;
  option dhcp-lease-time 259200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.247.48;
  option dhcp-renewal-time 84600;
  option dhcp-rebinding-time 207400;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:64:68:63:70:6c:69:76:65:62:6f:78:xxx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 0:0:5:58:c:1:a:0:1:0:0:0:ff:ff:ff:ff:ff;
  renew 1 2023/2/13 14:45:33;
  rebind 2 2023/2/14 08:23:03;
  expire 3 2023/2/15 15:15:33;
}
lease {
  interface "vlan0.832";
  fixed-address 83.xx.xx.96;
  next-server 80.10.247.48;
  option subnet-mask 255.255.248.0;
  option routers 83.xx.xx.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option domain-name "orange.fr";
  option broadcast-address 83.xx.xx.255;
  option dhcp-lease-time 259200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.247.48;
  option dhcp-renewal-time 84600;
  option dhcp-rebinding-time 207400;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:64:68:63:70:6c:69:76:65:62:6f:78:xxx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 0:0:5:58:c:1:a:0:1:0:0:0:ff:ff:ff:ff:ff;
  renew 1 2023/2/13 14:46:08;
  rebind 2 2023/2/14 08:23:38;
  expire 3 2023/2/15 15:16:08;
}
lease {
  interface "vlan0.832";
  fixed-address 83.xx.xx.96;
  next-server 80.10.247.175;
  option subnet-mask 255.255.248.0;
  option routers 83.xx.xx.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option domain-name "orange.fr";
  option broadcast-address 83.xx.xx.255;
  option dhcp-lease-time 259200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.247.175;
  option dhcp-renewal-time 77562;
  option dhcp-rebinding-time 207400;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:64:68:63:70:6c:69:76:65:62:6f:78:xxx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 0:0:5:58:c:1:a:0:1:0:0:0:ff:ff:ff:ff:ff;
  renew 4 2023/2/16 12:48:53;
  rebind 5 2023/2/17 04:58:21;
  expire 6 2023/2/18 15:16:11;
}
lease {
  interface "vlan0.832";
  fixed-address 83.xx.xx.96;
  next-server 80.10.247.175;
  option subnet-mask 255.255.248.0;
  option routers 83.xx.xx.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option domain-name "orange.fr";
  option broadcast-address 83.xx.xx.255;
  option dhcp-lease-time 259200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.247.175;
  option dhcp-renewal-time 79029;
  option dhcp-rebinding-time 207400;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:64:68:63:70:6c:69:76:65:62:6f:78:xxx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 0:0:5:58:c:1:a:0:1:0:0:0:ff:ff:ff:ff:ff;
  renew 5 2023/2/17 10:46:04;
  rebind 6 2023/2/18 03:13:54;
  expire 0 2023/2/19 12:48:55;
}
And the last renew was :

Code: [Select]

<27>1 2023-02-16T13:48:53+01:00 opnsense.local dhclient 22 - [meta sequenceId="1"] unknown dhcp option value 0x5a
<27>1 2023-02-16T13:48:53+01:00 opnsense.local dhclient 22 - [meta sequenceId="2"] unknown dhcp option value 0x7d
<27>1 2023-02-16T13:48:55+01:00 opnsense.local dhclient 22 - [meta sequenceId="3"] unknown dhcp option value 0x5a
<27>1 2023-02-16T13:48:55+01:00 opnsense.local dhclient 22 - [meta sequenceId="4"] unknown dhcp option value 0x7d
<13>1 2023-02-16T13:48:55+01:00 opnsense.local dhclient 25784 - [meta sequenceId="5"] New IP Address (vlan0.832): 83.xx.xx.96
<13>1 2023-02-16T13:48:55+01:00 opnsense.local dhclient 27363 - [meta sequenceId="6"] New Subnet Mask (vlan0.832): 255.255.248.0
<13>1 2023-02-16T13:48:55+01:00 opnsense.local dhclient 29780 - [meta sequenceId="7"] New Broadcast Address (vlan0.832): 83.xx.xx.255
<13>1 2023-02-16T13:48:55+01:00 opnsense.local dhclient 31258 - [meta sequenceId="8"] New Routers (vlan0.832): 83.xx.xx.1
<13>1 2023-02-16T13:48:55+01:00 opnsense.local dhclient 36866 - [meta sequenceId="9"] Creating resolv.conf
<13>1 2023-02-16T13:48:55+01:00 opnsense.local opnsense 53565 - [meta sequenceId="10"] /usr/local/etc/rc.newwanip: No IP change detected for WAN[wan]

I also have these files with the correct ip/values

Code: [Select]

# ls /tmp/vlan0.832_*
/tmp/vlan0.832_nameserver /tmp/vlan0.832_oldipv6 /tmp/vlan0.832_router
/tmp/vlan0.832_oldip /tmp/vlan0.832_prefixv6 /tmp/vlan0.832_searchdomain


[franco]

@skool, what are the last couple of lines before this one?

<27>1 2023-02-16T13:48:53+01:00 opnsense.local dhclient 22 - [meta sequenceId="1"] unknown dhcp option value 0x5a


thanks so far!

[skool]

@franco
I'm looking on /var/log/system/system_xxx.log file
previous lines are older and not related

Code: [Select]

<13>1 2023-02-16T11:50:39+01:00 opnsense.local flowd_aggregate.py 67247 - [meta sequence
Id="11"] vacuum interface_086400.sqlite
<13>1 2023-02-16T11:50:39+01:00 opnsense.local flowd_aggregate.py 67247 - [meta sequence
Id="12"] vacuum done
<27>1 2023-02-16T13:48:53+01:00 opnsense.local dhclient 22 - [meta sequenceId="1"] unknown dhcp option value 0x5a
<27>1 2023-02-16T13:48:53+01:00 opnsense.local dhclient 22 - [meta sequenceId="2"] unknown dhcp option value 0x7d

I can add locally some debug in files, if relevant.
But as I just renewed my IP (at 11:46:04), I need to wait for tomorrow to have new logs :/

[franco]

Thanks, the absence of further logs seems odd to me considering the default route is gone, but that doesn't mean too much. I would have just suspected the code tripping over itself here.

Once this bad state is entered, can you run the following to see if that fixes it?

# /usr/local/etc/rc.routing_configure

The lease information looks normal and how I would expect it to be fairly similar to test setups here.


Cheers,
Franco

[skool]

Hi Franco,

tested this morning :

Code: [Select]

root@opnsense:~ # route show default
route: route has not been found
root@opnsense:~ # /usr/local/etc/rc.routing_configure
Setting up routes...done.
Setting up gateway monitors...done.
Configuring firewall.......done.
root@opnsense:~ # route show default
   route to: default
destination: default
       mask: default
    gateway: lfbn-idf2-1-158-1.w83-202.abo.wanadoo.fr
        fib: 0
  interface: vlan0.832
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0


[franco]

So this brings back a working state until the next renew?


Cheers,
Franco