Gateway randomly lost

[Ben.]

Hi,
My OPNsense box loses its gateway randomly (can work for weeks, then suddenly lost).

System -> Gateways -> Single is empty in these cases, also the Dashboard shows an empty table for "Gateways".
In the logs I couldnt find anything special.

My box is connected to a cable modem, using the re driver.

It is the only gateway configured, no special settings have been changed.

• Upstream Gateway: checked
• Monitor IP: 1.1.1.3

The box is a Fujitsu Futro S930.

Do you have hints what I could check? In the logs I couldnt find anything specific.

I thought that it might be the cable to the modem, even it worked for years.
I didnt have this issue with my APU unit. Not sure if it's related to the hardware.

Thanks for hints.

[Ben.]

I found now these logs after it happened twice today:

Code: [Select]

2023-03-06T12:57:28 Critical dhclient exiting.
2023-03-06T12:57:28 Error dhclient connection closed
2023-03-06T12:57:28 Notice opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for wan(re0)
2023-03-06T08:24:57 Notice flowd_aggregate.py vacuum done
2023-03-06T08:24:57 Notice flowd_aggregate.py vacuum interface_086400.sqlite
So the WAN just detaches.
Nothing helps but a reboot.

First attempt to fix it: replace the WAN cable

[Ben.]

Replacing the cable did not help.

The re0 interface is there but IP is 0.0.0.0

Can I reset the interface without a reboot?
Any idea what I could check?

[sorano]

My first fix would be to replace the NIC with an Intel one.

Realtek are well known for misbehaving randomly and causing problems.

There's a reason you never see Realtek NICs in high-end server hardware.

[Ben.]

This is the next step then.

I thought the S930 is a popular device for OPNsense/pfSense and should work quite well.

Thanks for the hint.

[Ben.]

Before I switch the NIC I wanted to share this from the logs:

Code: [Select]

<13>1 2023-03-10T00:00:06+01:00 host.domain.com rule-updater.py 51274 - [meta sequenceId="9"] download completed for https://threatfox.abuse.ch/downloads/threatfox_suricata.rules
<11>1 2023-03-10T00:42:03+01:00 host.domain.com opnsense 78174 - [meta sequenceId="1"] /usr/local/etc/rc.dyndns: Dynamic DNS: updatedns() starting
<11>1 2023-03-10T00:42:03+01:00 host.domain.com opnsense 78174 - [meta sequenceId="2"] /usr/local/etc/rc.dyndns:
<11>1 2023-03-10T00:42:03+01:00 host.domain.com opnsense 78174 - [meta sequenceId="3"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): x.x.x.x extracted
<11>1 2023-03-10T00:42:03+01:00 host.domain.com opnsense 78174 - [meta sequenceId="4"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): running dyndns_failover_interface for wan. found re0
<11>1 2023-03-10T00:42:03+01:00 host.domain.com opnsense 78174 - [meta sequenceId="5"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): x.x.x.x extracted
<11>1 2023-03-10T00:42:03+01:00 host.domain.com opnsense 78174 - [meta sequenceId="6"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): Current WAN IP: x.x.x.x Cached IP: x.x.x.x
<11>1 2023-03-10T00:42:03+01:00 host.domain.com opnsense 78174 - [meta sequenceId="7"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
<13>1 2023-03-10T00:42:05+01:00 host.domain.com opnsense 81364 - [meta sequenceId="8"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for wan(re0)
<27>1 2023-03-10T00:42:05+01:00 host.domain.com dhclient 25327 - [meta sequenceId="9"] connection closed
<26>1 2023-03-10T00:42:05+01:00 host.domain.com dhclient 25327 - [meta sequenceId="10"] exiting.
<13>1 2023-03-10T00:42:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="11"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for wan(re0)
<11>1 2023-03-10T00:44:09+01:00 host.domain.com configctl 21089 - [meta sequenceId="1"] error in configd communication  Traceback (most recent call last):   File "/usr/local/sbin/configctl", line 66, in exec_config_cmd     line = sock.recv(65536).decode() socket.timeout: timed out
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 49149 - [meta sequenceId="2"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for wan(re0)
<27>1 2023-03-10T00:44:09+01:00 host.domain.com dhclient 24421 - [meta sequenceId="3"] connection closed
<26>1 2023-03-10T00:44:09+01:00 host.domain.com dhclient 24421 - [meta sequenceId="4"] exiting.
<11>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="5"] /usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dhclient.re0.pid' 're0'' returned exit code '15', the output was 'DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 re0 link state up -> down re0 link state down -> up DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 re0 link state up -> down DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 2 re0 link state down -> up DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 5 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 14 re0 link state up -> down DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 1 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 1 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 1 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 2 re0 link state down -> up DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 5 DHCPREQUEST on re0 to 255.255.255.255 port 67 re0 link state up -> down DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 14 re0 link state down -> up DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 11 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 2 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 2 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 5 re0 link state up -> down DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 14 re0 link state down -> up DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 re0 link state up -> down DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 6 DHCPREQUEST on re0 to 255.255.255.255 port 67 re0 link state down -> up DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 4 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 12 DHCPREQUEST on re0 to 255.255.255.255 port 67 re0 link state up -> down re0 link state down -> up DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 2 DHCPREQUEST on re0 to 255.255.255.255 port 67 DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 12 DHCPREQUEST on re0 to 255.255.255.255 port 67'
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="6"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="7"] /usr/local/etc/rc.linkup: plugins_configure monitor (,WAN_DHCP)
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="8"] /usr/local/etc/rc.linkup: plugins_configure monitor (execute task : dpinger_configure_do(,WAN_DHCP))
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="9"] /usr/local/etc/rc.linkup: plugins_configure ipsec (,wan)
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="10"] /usr/local/etc/rc.linkup: plugins_configure ipsec (execute task : ipsec_configure_do(,wan))
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="11"] /usr/local/etc/rc.linkup: plugins_configure dhcp ()
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: plugins_configure dns ()
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="14"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2023-03-10T00:44:09+01:00 host.domain.com opnsense 21250 - [meta sequenceId="15"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : unbound_configure_do())
<13>1 2023-03-10T00:44:11+01:00 host.domain.com opnsense 93367 - [meta sequenceId="16"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for wan(re0)
<11>1 2023-03-10T00:44:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="17"] /usr/local/etc/rc.dyndns: Dynamic DNS: updatedns() starting
<11>1 2023-03-10T00:44:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="18"] /usr/local/etc/rc.dyndns:
<11>1 2023-03-10T00:44:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="19"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): 0.0.0.0 extracted
<11>1 2023-03-10T00:44:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="20"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): running dyndns_failover_interface for wan. found re0
<11>1 2023-03-10T00:44:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="21"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): 0.0.0.0 extracted
<11>1 2023-03-10T00:44:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="22"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): Current WAN IP: 0.0.0.0 Cached IP: x.x.x.x
<11>1 2023-03-10T00:44:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="23"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): Dynamic DNS: cacheIP != wan_ip.  Updating. Cached IP: x.x.x.x WAN IP: 0.0.0.0
<11>1 2023-03-10T00:44:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="24"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com via Cloudflare API token): _update() starting.
<11>1 2023-03-10T00:45:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="1"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): _checkStatus() starting.
<11>1 2023-03-10T00:45:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="2"] /usr/local/etc/rc.dyndns: Dynamic DNS (host.domain.com): Current Service: cloudflare-token
<11>1 2023-03-10T00:45:11+01:00 host.domain.com opnsense 1644 - [meta sequenceId="3"] /usr/local/etc/rc.dyndns: Curl error occurred: Resolving timed out after 15002 milliseconds
<27>1 2023-03-10T00:45:48+01:00 host.domain.com dhclient 96956 - [meta sequenceId="4"] send_packet: Network is down
<27>1 2023-03-10T00:46:01+01:00 host.domain.com dhclient 96956 - [meta sequenceId="5"] send_packet: Network is down
<11>1 2023-03-10T00:46:10+01:00 host.domain.com configctl 92916 - [meta sequenceId="6"] error in configd communication  Traceback (most recent call last):   File "/usr/local/sbin/configctl", line 66, in exec_config_cmd     line = sock.recv(65536).decode() socket.timeout: timed out
<13>1 2023-03-10T00:46:11+01:00 host.domain.com opnsense 69584 - [meta sequenceId="7"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for wan(re0)
<27>1 2023-03-10T00:46:11+01:00 host.domain.com dhclient 96956 - [meta sequenceId="8"] connection closed
<26>1 2023-03-10T00:46:11+01:00 host.domain.com dhclient 96956 - [meta sequenceId="9"] exiting.
<11>1 2023-03-10T00:46:11+01:00 host.domain.com opn
It looks like the legacy DynDNS script is causing an issue here, no?

[Ben.]

Seems disabling the DynDNS client fixed it.

Since then no issues.