WAN DHCP - Loss of IP no retry

[flashkiller]

Hello,

I recently set up a OPNSense system and everything worked great, but today the WAN interface lost its DHCP-IP.
The modem should give out a new IP as I have no static IP on the WAN interface. But for some reason this failed, I was able to solve the issue with reapplying the DHCP setting on the interface.

I also checked the log and will attach it below.
It seems as the WAN interface went down and came back up, but dhclient failed (error code 15) to gain a new IP.
I can understand that it could fail (nothing is perfect), but why does it not retry after some time instead of being stuck with no IP.

Is there a way to tell the dhclient to retry after one failure?

Thanks in advance

Code: [Select]

Jul  2 10:52:43 Jun kernel: igb0: link state changed to DOWN
Jul  2 10:52:44 Jun opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Jul  2 10:52:44 Jun opnsense: /usr/local/etc/rc.linkup: Clearing states to old gateway 84.112.13.1.
Jul  2 10:52:47 Jun kernel: igb0: link state changed to UP
Jul  2 10:52:48 Jun opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Jul  2 10:52:48 Jun opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Jul  2 10:53:03 Jun kernel: ovpnc1: link state changed to DOWN
Jul  2 10:53:11 Jun kernel: igb0: link state changed to DOWN
Jul  2 10:53:14 Jun kernel: igb0: link state changed to UP
Jul  2 10:54:28 Jun opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'igb0'
Jul  2 10:54:28 Jun opnsense: /usr/local/etc/rc.newwanip: On (IP address: 192.168.100.3) (interface: WAN[wan]) (real interface: igb0).
Jul  2 10:54:29 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Jul  2 10:54:29 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv4 default gateway set, assuming wan
Jul  2 10:54:29 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Jul  2 10:54:29 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 192.168.100.1
Jul  2 10:54:29 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '192.168.100.1'
Jul  2 10:54:29 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Jul  2 10:54:34 Jun opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Jul  2 10:54:43 Jun opnsense: /usr/local/etc/rc.newwanip: Dynamic DNS: updatedns() starting
Jul  2 10:54:47 Jun configd_ctl.py: error in configd communication  Traceback (most recent call last):   File "/usr/local/opnsense/service/configd_ctl.py", line 65, in exec_config_cmd     line = sock.recv(65536) timeout: timed out
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dhclient.igb0.pid' 'igb0'' returned exit code '15', the output was 'dhclient: PREINIT dhclient: Starting delete_old_states() dhclient: Comparing IPs: Old: 84.1.13.244 New: dhclient: Remo
ving states from old IP '84.1.13.244' (new IP '') 0 states cleared killed 0 src nodes from 1 sources and 0 destinations DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPDIS
COVER on igb0 to 255.255.255.255 port 67 interval 2 DHCPDISCOVER on igb0 to 255.255.255.255 port 67 interval 2 DHCPDISCOVER on igb0 to 255.255.255.255 port 67 interval 4 igb0 link state up -> down DHCPDISCOVER on igb0 to 255.255.255.255 port 67 interval 8 igb0 link state down -> up DHCPREQUEST on igb0 to 255.255.255
.255 port 67 DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPREQUEST o
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: no IPv4 default gateway set, assuming wan
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: no IPv6 default gateway set, assuming wan
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 192.168.100.1
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: creating /tmp/igb0_defaultgw using '192.168.100.1'
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: The command '/sbin/route add -'inet' default '192.168.100.1'' returned exit code '1', the output was 'route: writing to routing socket: Network is unreachable add net default: gateway 192.168.100.1 fib 0: Network is unreachable'
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
Jul  2 10:54:48 Jun opnsense: /usr/local/etc/rc.linkup: Clearing states to old gateway 192.168.100.1.
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dhclient.igb0.pid' 'igb0'' returned exit code '1', the output was 'dhclient already running, pid: 30344. exiting.'
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: no IPv4 default gateway set, assuming wan
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: no IPv6 default gateway set, assuming wan
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.linkup: ROUTING: skipping IPv6 default route
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.newwanip: Aborted IPv4 detection: Resolving timed out after 5722 milliseconds
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.newwanip: Dynamic DNS (@.flk-industries.com): IP address could not be extracted
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.newwanip: Dynamic DNS (@.flk-industries.com): running get_failover_interface for wan. found igb0
Jul  2 10:54:49 Jun opnsense: /usr/local/etc/rc.newwanip: Dynamic DNS (@.flk-industries.com) There was an error trying to determine the public IP for interface - wan(igb0). Probably interface is not a WAN interface.
Jul  2 10:54:51 Jun opnsense: /usr/local/etc/rc.newwanip: Aborted IPv4 detection: no address for igb0
Jul  2 10:54:51 Jun opnsense: /usr/local/etc/rc.newwanip: Dynamic DNS (wien.flk-industries.com) There was an error trying to determine the public IP for interface - wan(igb0). Probably interface is not a WAN interface.
Jul  2 10:54:52 Jun opnsense: /usr/local/etc/rc.newwanip: Aborted IPv4 detection: no address for igb0
Jul  2 10:54:52 Jun opnsense: /usr/local/etc/rc.newwanip: Dynamic DNS (acme.flk-industries.com) There was an error trying to determine the public IP for interface - wan(igb0). Probably interface is not a WAN interface.
Jul  2 10:54:53 Jun opnsense: /usr/local/etc/rc.linkup: The command '/usr/local/opnsense/scripts/dns/unbound_dhcpd.py /domain 'flk'' returned exit code '1', the output was 'Unable to lock on the pidfile.'
Jul  2 10:54:55 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS: updatedns() starting
Jul  2 10:54:55 Jun opnsense: /usr/local/etc/rc.dyndns: Aborted IPv4 detection: no address for igb0
Jul  2 10:54:55 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (@.flk-industries.com): IP address could not be extracted
Jul  2 10:54:55 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (@.flk-industries.com): running get_failover_interface for wan. found igb0
Jul  2 10:54:55 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (@.flk-industries.com) There was an error trying to determine the public IP for interface - wan(igb0). Probably interface is not a WAN interface.
Jul  2 10:54:56 Jun opnsense: /usr/local/etc/rc.newwanip: The command '/usr/local/opnsense/scripts/dns/unbound_dhcpd.py /domain 'flk'' returned exit code '1', the output was 'Unable to lock on the pidfile.'
Jul  2 10:54:56 Jun opnsense: /usr/local/etc/rc.newwanip: The command '/usr/local/sbin/unbound -c '/var/unbound/unbound.conf'' returned exit code '1', the output was '[1530521696] unbound[8152:0] error: bind: address already in use [1530521696] unbound[8152:0] fatal error: could not open ports'
Jul  2 10:54:57 Jun opnsense: /usr/local/etc/rc.dyndns: Aborted IPv4 detection: no address for igb0
Jul  2 10:54:57 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (wien.flk-industries.com) There was an error trying to determine the public IP for interface - wan(igb0). Probably interface is not a WAN interface.
Jul  2 10:54:57 Jun opnsense: /usr/local/etc/rc.linkup: The command '/usr/local/sbin/unbound -c '/var/unbound/unbound.conf'' returned exit code '1', the output was '[1530521697] unbound[25131:0] error: bind: address already in use [1530521697] unbound[25131:0] fatal error: could not open ports'
Jul  2 10:54:59 Jun opnsense: /usr/local/etc/rc.dyndns: Aborted IPv4 detection: no address for igb0
Jul  2 10:54:59 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (acme.flk-industries.com) There was an error trying to determine the public IP for interface - wan(igb0). Probably interface is not a WAN interface.
Jul  2 10:54:59 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS: updatedns() starting
Jul  2 10:54:59 Jun opnsense: /usr/local/etc/rc.dyndns: Aborted IPv4 detection: no address for igb0
Jul  2 10:54:59 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (@.flk-industries.com): IP address could not be extracted
Jul  2 10:54:59 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (@.flk-industries.com): running get_failover_interface for wan. found igb0
Jul  2 10:54:59 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (@.flk-industries.com) There was an error trying to determine the public IP for interface - wan(igb0). Probably interface is not a WAN interface.
Jul  2 10:55:01 Jun opnsense: /usr/local/etc/rc.dyndns: Aborted IPv4 detection: no address for igb0
Jul  2 10:55:01 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (wien.flk-industries.com) There was an error trying to determine the public IP for interface - wan(igb0). Probably interface is not a WAN interface.
Jul  2 10:55:02 Jun opnsense: /usr/local/etc/rc.dyndns: Aborted IPv4 detection: no address for igb0
Jul  2 10:55:02 Jun opnsense: /usr/local/etc/rc.dyndns: Dynamic DNS (acme.flk-industries.com) There was an error trying to determine the public IP for interface - wan(igb0). Probably interface is not a WAN interface.
Jul  2 16:00:38 Jun opnsense: /index.php: Successful login for user 'admin' from: 192.168.10.121
Jul  2 16:01:39 Jun opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'igb0'
Jul  2 16:01:39 Jun opnsense: /usr/local/etc/rc.newwanip: On (IP address: 84.1.13.244) (interface: WAN[wan]) (real interface: igb0).
Jul  2 16:01:40 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Jul  2 16:01:40 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv4 default gateway set, assuming wan
Jul  2 16:01:40 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Jul  2 16:01:40 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 84.112.13.1
Jul  2 16:01:40 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '84.112.13.1'
Jul  2 16:01:40 Jun opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Jul  2 16:01:45 Jun opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.


[flashkiller]

OK, found it, in the interface menu under "DHCP client configuration" there are the settings.

It is just interesting that they are empty by default?
See attachment.

Below this part are even some presets, I now selected "OPNSense Default".

That should probably fix this issue.

[franco]

Hi there,

Not sure it will fix it. This usually happens when you have a modem in front of OPNsense and it takes unusually long to recover (ISP maintenance for example). You have no link up/down on the upstream link from your OPNsense so at some point DHCP just gives up.

What you can do is enable gateway monitoring to let this restart automatically if it detects a dead link.

For this ping a target beyond your gateway, most people use 8.8.8.8 to be sure it reaches the Internet.


Cheers,
Franco

[flashkiller]

Hello Franco,

thanks for the reply.
What you are saying could very well be the case in my environment as there is a modem in front that is provided by my Provider, so maybe the changed something on it.

In regards to "gateway monitoring", do you mean that I just enter a pingable IP (eg 8.8.8. under System-Gateway-Single-Monitoring IP? Or did you mean something different?
I do not quiet understand what that does. Could you elaborate?
How does this cause the interface to retry getting a new Lease?
The failure in this one case was only for a few seconds (see log), why would these settings (retry-count under the interface) not work?

Sorry for asking these questions, but I am just starting out with OPNSense and would like to understand this better.

Thanks

[franco]

Hi flashkiller,

Yes, you are on the right track. Go to your IPv4 default gateway under System: Gateways: Single and edit it, add e.g. 8.8.8.8 as the monitor IP and uncheck "Disable Gateway Monitoring". Hit "Save" and then on the next page "Apply changes".

That will start the monitoring service. It polls the monitor and will generate events when the Internet is down and automatically restart if it's back up.


Cheers,
Franco

[flashkiller]

Hi Franco,

thanks, I set it up as you said.

But am still somewhat confused:
In case the same problem occures again, WAN losing its IP and having none, how does this setting remediate the issue?

Cheers

[franco]

Now you got me. I'm not sure if this is solely interesting for Multi-WAN links that have other types connectivity issues.

But there is potential for a new feature that keeps reloading the interface until it comes back up in a sane periodic interval, let's say 10 to 15 minutes as long as the link seems down?


Cheers,
Franco

[flashkiller]

I read the documentation about that part with Gateway monitoring.
And like you said I mostly found information about a Multi-WAN setup.

A feature like you said would be a possibility, but in search of something like it I was not successful.
There would probably a possibility of writing a cronjob that checks if the WAN-Interface has an IP and if it does not it could possible reload the interface.
Problem on my part is that I do not want to divert to much from what is possible with the Webinterface.

A plugin that would be interesting is a cronjob-manager, that would allow to manage these on the Webinterface.

Maybe someone in the future will write something like that.

For now thanks for your help in clearing things up.

[franco]

I've added a reminder ticket: https://github.com/opnsense/core/issues/2517


Cheers,
Franco

[flashkiller]

Thanks, hope to see this soon in production.