lost wan ip

[hboetes]

Yesterday I was called out of bed because one of our VPNs was down. When I finally got access to the remote OPNsense box, with the help of a junior support engineer who was on site, I discovered the WAN interface had no external IP. So I pressed the save button to restart dhclient and lo and behold the interface came up again.

I just checked the log output and it sort of baffles me. Have a look for yourself.


# strings system.log | sort | grep dhclient
(SNIP)
Nov 18 09:16:47 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:17:01 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:17:01 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:17:18 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:17:18 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:17:56 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:17:56 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:19:24 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:19:24 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:21:52 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:21:52 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:22:08 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:22:08 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:22:22 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:22:22 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:22:50 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:22:50 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:23:32 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:23:32 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:24:25 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:24:25 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:25:52 kiwifw dhclient[3610]: send_packet: No route to host
Nov 18 09:25:52 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 09:27:43 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Nov 18 09:29:18 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Nov 18 09:32:30 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Nov 18 09:34:27 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Nov 18 09:37:39 kiwifw dhclient[43029]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Nov 18 09:39:03 kiwifw dhclient[3610]: connection closed
Nov 18 09:39:03 kiwifw dhclient[3610]: exiting.
Nov 18 09:39:04 kiwifw dhclient: Comparing IPs: Old: 192.168.2.116 New: 
Nov 18 09:39:04 kiwifw dhclient: Comparing IPs: Old: 192.168.2.116 New: 192.168.2.116
Nov 18 09:39:04 kiwifw dhclient: Creating resolv.conf
Nov 18 09:39:04 kiwifw dhclient: New Broadcast Address (igb1): 192.168.2.255
Nov 18 09:39:04 kiwifw dhclient: New IP Address (igb1): 192.168.2.116
Nov 18 09:39:04 kiwifw dhclient: New Routers (igb1): 192.168.2.1
Nov 18 09:39:04 kiwifw dhclient: New Subnet Mask (igb1): 255.255.255.0
Nov 18 09:39:04 kiwifw dhclient: Removing states from old IP '192.168.2.116' (new IP '')
Nov 18 09:39:04 kiwifw dhclient: Starting delete_old_states()
Nov 18 09:39:04 kiwifw dhclient: Starting delete_old_states()
Nov 18 09:39:04 kiwifw dhclient: route add default 192.168.2.1
Nov 18 09:39:04 kiwifw dhclient[57449]: DHCPACK from 192.168.2.1
Nov 18 09:39:04 kiwifw dhclient[57449]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
Nov 18 09:39:07 kiwifw dhclient[57449]: bound to 192.168.2.116 -- renewal in 43200 seconds.
Nov 18 21:39:04 kiwifw dhclient: Creating resolv.conf
Nov 18 21:39:04 kiwifw dhclient[7973]: DHCPACK from 192.168.2.1
Nov 18 21:39:04 kiwifw dhclient[7973]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 18 21:39:04 kiwifw dhclient[7973]: bound to 192.168.2.116 -- renewal in 43200 seconds.
Nov 19 09:39:04 kiwifw dhclient: Creating resolv.conf
Nov 19 09:39:04 kiwifw dhclient[7973]: DHCPACK from 192.168.2.1
Nov 19 09:39:04 kiwifw dhclient[7973]: DHCPREQUEST on igb1 to 192.168.2.1 port 67
Nov 19 09:39:04 kiwifw dhclient[7973]: bound to 192.168.2.116 -- renewal in 43200 seconds.


Apparently the provider does something to the network at night, I can't tell what.

I like the flexibility of DHCP ensuring that whatever the external provider changes things keep working. But in this case dhclient wasn't very resilient in keeping the connection working.

What are your thoughts on the matter?

[hboetes]

• According to the local network engineer there was a power outage, because of the UPS our devices didn't shut down.
• dhclient kept asking for an ip, even though the dhcpserver was already up it kept reporting it had no route to it.
• Restarting dhclient(by pressing the save button)got dhclient working again.

[l8gravely]

It would still be nice if it was a bit more robust on the WAN port.  My ISP (Charter) does something stupid and my APU1 (now APU4d4) running 20.7.<latest> keeps dropping the WAN connection randomly.  So the kids/wife yell and I scurry off to hit reload on the Interfaces page.  Total pain.  It *never* used to do this with 19.x or earlier releases, but I also suspect my ISP is under alot more load now that we're all home.

But.... how can I make it more robust?  Or how can I log more and better data to see what's actually happening?  That's the big question. 

[hboetes]

I can think of 2 solutions, you can explain your family to reconnect the external cable after the connection got restored.

You could also set up a cronjob which checks internet connection with a ping to the gateway and restarts if it's unreachable.

[marjohn56]

That's exactly what I have, a Cron script that runs every few minutes and pings upstream. If it gets no response after a minute ( or two.. can't remember ) it will try a WAN down/up, if still no go after another minute then it will reboot.