DHCP lease not renewing on Orange FR

[skool]

Hello,

Since ~10 days, without doing anything on the router, we are multiple opnsense/pfsense users having the same issue with Orange in France. Every ~24h, our internet connection drops on IPv4/IPv6.
On a IP point of view, on opnsense, all is good (routes, interfaces, etc…) but the gateway dont ping.
We need to manually restart dhclient (by unplug/replug cable, or restarting interface, or …)

I had done it yesterday at 18:40. just after that, my dhcp lease looked like that :

Code: [Select]

lease {
  interface "vlan0.832";
  fixed-address 83.202.25.xx;
  next-server 80.10.234.173;
  option subnet-mask 255.255.248.0;
  option routers 83.202.24.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option host-name "opnsense";
  option broadcast-address 83.202.31.255;
  option dhcp-lease-time 604800;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.234.173;
  option dhcp-renewal-time 84672;
  option dhcp-rebinding-time 483840;
  option dhcp-client-identifier 1:ac:84:c9:xx:xx:xx;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx;
  renew 1 2023/4/3 16:34:08;
  rebind 2 2023/4/4 10:12:32;
  expire 0 2023/4/9 17:02:56;
}
lease {
  interface "vlan0.832";
  fixed-address 83.202.25.xx;
  next-server 80.10.234.173;
  option subnet-mask 255.255.248.0;
  option routers 83.202.24.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option host-name "opnsense";
  option broadcast-address 83.202.31.255;
  option dhcp-lease-time 604800;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.234.173;
  option dhcp-renewal-time 84672;
  option dhcp-rebinding-time 483840;
  option dhcp-client-identifier 1:ac:84:c9:xx:xx:xx;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx;
  renew 1 2023/4/3 16:34:20;
  rebind 2 2023/4/4 10:12:44;
  expire 0 2023/4/9 17:03:08;
}
(dhcp lease is present twice on the file)

Today, my system.log contained this log lines :

Code: [Select]

<27>1 2023-04-03T18:54:37+02:00 opnsense.local dhclient 77948 - [meta sequenceId="1"] send_packet: No route to host
<27>1 2023-04-03T18:56:29+02:00 opnsense.local dhclient 77948 - [meta sequenceId="1"] send_packet: No route to host
<27>1 2023-04-03T18:56:57+02:00 opnsense.local dhclient 77948 - [meta sequenceId="2"] send_packet: No route to host
<27>1 2023-04-03T18:57:19+02:00 opnsense.local dhclient 77948 - [meta sequenceId="3"] send_packet: No route to host
<27>1 2023-04-03T18:57:50+02:00 opnsense.local dhclient 77948 - [meta sequenceId="4"] send_packet: No route to host
<27>1 2023-04-03T18:58:27+02:00 opnsense.local dhclient 77948 - [meta sequenceId="5"] send_packet: No route to host
<27>1 2023-04-03T18:59:10+02:00 opnsense.local dhclient 77948 - [meta sequenceId="6"] send_packet: No route to host
<27>1 2023-04-03T19:00:16+02:00 opnsense.local dhclient 77948 - [meta sequenceId="1"] send_packet: No route to host
<27>1 2023-04-03T19:02:32+02:00 opnsense.local dhclient 77948 - [meta sequenceId="1"] send_packet: No route to host
<27>1 2023-04-03T19:05:53+02:00 opnsense.local dhclient 77948 - [meta sequenceId="1"] send_packet: No route to host
<27>1 2023-04-03T19:12:47+02:00 opnsense.local dhclient 77948 - [meta sequenceId="1"] send_packet: No route to host
<27>1 2023-04-03T19:20:33+02:00 opnsense.local dhclient 77948 - [meta sequenceId="1"] send_packet: No route to host
<27>1 2023-04-03T19:30:19+02:00 opnsense.local dhclient 77948 - [meta sequenceId="1"] send_packet: No route to host

until I was back at home and restarted it, at 19:34 today.

Now, my lease file is

Code: [Select]

lease {
  interface "vlan0.832";
  fixed-address 83.202.25.xx;
  next-server 80.10.234.173;
  option subnet-mask 255.255.248.0;
  option routers 83.202.24.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option host-name "opnsense";
  option broadcast-address 83.202.31.255;
  option dhcp-lease-time 604800;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.234.173;
  option dhcp-renewal-time 84672;
  option dhcp-rebinding-time 483840;
  option dhcp-client-identifier 1:ac:84:c9:xx:xx:xx;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx;
  renew 1 2023/4/3 16:34:20;
  rebind 2 2023/4/4 10:12:44;
  expire 0 2023/4/9 17:03:08;
}
lease {
  interface "vlan0.832";
  fixed-address 83.202.25.xx;
  next-server 80.10.234.173;
  option subnet-mask 255.255.248.0;
  option routers 83.202.24.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option host-name "opnsense";
  option broadcast-address 83.202.31.255;
  option dhcp-lease-time 604800;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.234.173;
  option dhcp-renewal-time 70604;
  option dhcp-rebinding-time 483840;
  option dhcp-client-identifier 1:ac:84:c9:xx:xx:xx;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx;
  renew 2 2023/4/4 13:10:56;
  rebind 3 2023/4/5 03:53:29;
  expire 1 2023/4/10 17:34:12;
}

I dont know where to look at to debug what happened.

If someone having an idea…
Many thanks

[skool]

Today, when my DHCP lease expired, I removed /var/db/dhclient.lease file before renewing IP

To renew, I used GUI, ask for a release before a renew.
And now, the dhclient.lease file contains only 1 lease that seems correct.
I will see tomorrow at 17:15 if something changes.

[skool]

And tomorrow, same issue…
As I see on /var/db/dhclient.leases… the client added a 2nd lease when manually renew.

That's my new dhclient.lease file :

Code: [Select]

lease {
  interface "vlan0.832";
  fixed-address 83.202.25.xx;
  next-server 80.10.234.173;
  option subnet-mask 255.255.248.0;
  option routers 83.202.24.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option host-name "opnsense";
  option broadcast-address 83.202.31.255;
  option dhcp-lease-time 604800;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.234.173;
  option dhcp-renewal-time 93695;
  option dhcp-rebinding-time 483840;
  option dhcp-client-identifier 1:ac:84:c9:da:93:40;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:xxxxxxxxxxxxx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 0:0:5xxxxxxxxx;
  renew 3 2023/4/5 15:15:14;
  rebind 4 2023/4/6 10:46:20;
  expire 2 2023/4/11 13:13:39;
}
lease {
  interface "vlan0.832";
  fixed-address 83.202.25.xx;
  next-server 80.10.234.173;
  option subnet-mask 255.255.248.0;
  option routers 83.202.24.1;
  option domain-name-servers 80.10.246.1,81.253.149.9;
  option host-name "opnsense";
  option broadcast-address 83.202.31.255;
  option dhcp-lease-time 604800;
  option dhcp-message-type 5;
  option dhcp-server-identifier 80.10.234.173;
  option dhcp-renewal-time 88197;
  option dhcp-rebinding-time 483840;
  option dhcp-client-identifier 1:ac:84:c9:da:93:40;
  option option-90 0:0:0:0:0:0:0:0:0:0:0:xxxxxxxxxxxxx;
  option domain-search "MSR.access.orange-multimedia.net.";
  option option-125 0:0:5xxxxxxxxx;
  renew 4 2023/4/6 16:12:39;
  rebind 5 2023/4/7 10:35:05;
  expire 3 2023/4/12 15:42:42;
}

The two leases are exactly the same except the renew dates and renewal times.

Is there a way to enable more verbose on dhclient, to see what Orange answers when we made a dhclient query ?
And is it normal to have 2 leases on the file ?

[nivek1612]

Yes lots of people with this issue. I have been okay until yesterday when I lost my connection
It may not be the same issue as it’s less than 24 hours since I rebooted the router. Time will tell

I think something has changed at Orange as like you I didn’t  change anything

@franco which logs should we capture

[nivek1612]

Confirming my connection no longer renews I’m on 23.1.5_4

The french forum suggest that this issue is not present in 22.7
Is there an easy way to revert so I can test

[nivek1612]

Seems the issue has appeared since 23.1.3_4-amd64

People on that version are ok

What is the process to revert, is this the correct syntax

opnsense-revert -r 23.1.3 opnsense

[Sisyphe]

Same issue for me, it seems the dhclient is not renewing the IPv4 once the lease has expired.

[skool]

If it helps for debugging, I started a packet capture of all DHCP (v4 and v6) trames.
I see the initial query and answer, but on the packets, all seems to be correct.

[nivek1612]

I have reverted to 23.1.3
Lets see if the renewal works

[franco]

I'm not sure we have substantial changes here, especially when pfSense users are reporting the same problem.

I'm not at the office this week. Not sure how to proceed as well. Even packet captures might be useless if the other side simply refuses to respond. In that case a packet capture from their router would be the only choice...


Cheers,
Franco

[nivek1612]

I’m not back in France for a few months
I’m connecting to the router over VPN so can’t swap the livebox in to test

@skool & @sisyphe are you able to connect the livebox and capture the packets at renewal

[skool]

Hi,

I looked at what happened when renew tested, having an eye on lease file and on packet capture.

Before renewing, I had a lease file with 2 blocs, identical except dates. One lease had a renew date at this morning, a second at 13:35:30 UTC

On my capture, before 13:35:30 UTC, nothing else that the initial dhcp request yesterday. No packet were sent or received (that seems normal)

Today at 13:35:30 UTC exactly, I see a dhcp request sent to the previous DHCP server, but no response. This packet is resent every ~10s, but no response.

At this time, the lease file didnt change.

On the GUI, i clicked on the Renew button (without trying to release before), and I see 3 dhcp packets sent to broadcast. The 2 first are flagged as DHCP Requests by Wireshark, the 3rd is a « DHCP Discover », and after this 3rd packet, I see a response from my ISP with a « DHCP Offer » packet.
I see a final exchange between my router and the dhcp server before my connection goes UP again.

Between 13:35:30 UTC and the last dhcp response, my internet acces was down.

On the lease file, when I clicked on the renew button, the first lease disappeared (the one with a renew date at this morning), and after the final dhcp exchange, I now have 2 leases : the one that was renewed today, and a new one with a renew date for tomorrow.


So, my first supposition is wrong. the client is sending a DHCP request when needed, the problem seems that the DHCP server dont received/answered to it.



@nivek1612: I dont have harware to capture the renew packets between the livebox and Orange dhcp server.

edit: I re-read what the Orange guy said on the french forum, the DHCP on v4 need to follow the lifecycle DORA, and as I understand, it's a « discover, offer, request, ack »
it's what I viewed on the capture when manually renewed my IP
but the automatic way is only a Request, not preceded by the discovery/offer exchange.

my new suppositition is :
- Orange see a first dhcp request for the renew
- it's not what expected (he want a Discover)
- Orange drop the connection
- when renewing manually, we respect the DORA lifecycle, all back to normal

it's the same thing on DHCPv6. the first requests was DHCP Renew, not Solicit, but when we manually renew, it send a solicit, receives an Advertisement, send a Request, and receive a Reply (SARR lifecycle)


edit2:
following man page of dhclient, it's normal it keeps all dhcp leases that are not expired.
if I unplug/replug the cable, the old lease is kept, but as it's not correctly renewed, a new one is requested, it's why we have 2 leases on the db file.

[skool]

So, after reading some docs, I suppose the DORA cycle is not a problem.
But maybe the vlan-pcp option is not correctly used on renew packets. It's mandatory for Orange to set a priority of 6 on every requests.
My capture is made before vlan tagging, so I cant see if it's set or not.

But I can confirm that the initial DHCP packet is correctly tagged with a priority of 6. the default vlan priority is set to 0.

Edit: I added a pf rule to set priority to 6 for all dhcp outgoing queries, to see if it changes something…

[nivek1612]

Looking forward to seeing the outcome of you tests. Being remote I can’t play around too much

[skool]

I confirm that adding a firewall rule to re-tag priority to 6 for DHCP packets (outgoing UDP packets to 67 and 546, ipv4 and ipv6) fix the issue.

As I read on other forum, it seems this problem is also present on Mikrotik equipment, that probably use the same dhclient app.

So the bug is that dhclient dont use vlan-pcp 6 when renewing a lease.
I dont know if someone is possible on opnsense side or if we need to check with dhclient team.