WAN is suddenly down for a minute - recurring after hours

Started by Bytechanger, February 12, 2022, 11:41:36 PM

Previous topic - Next topic
Print
Go Down Pages1
User actions
February 12, 2022, 11:41:36 PM
Hi,

my OPNSense runs good on my proxmox server.
But now I see recurring WAN breaks after hours. In proxmox-Log there is nothing but on OPNSense it seems it is a "ip renew" but it is the old ip.

it seems to be a dhcp update and something is going wrong in opnsense

Code Select

2022-02-12T19:57:39 Notice opnsense plugins_configure newwanip (execute task : webgui_configure_do(,opt4))
2022-02-12T19:57:39 Notice opnsense plugins_configure newwanip (execute task : vxlan_configure_interface())
2022-02-12T19:57:39 Notice opnsense plugins_configure newwanip (execute task : unbound_configure_do(,opt4))
2022-02-12T19:57:39 Notice opnsense plugins_configure newwanip (execute task : openssh_configure_do(,opt4))
2022-02-12T19:57:39 Notice opnsense plugins_configure newwanip (execute task : opendns_configure_do())
2022-02-12T19:57:39 Notice opnsense plugins_configure newwanip (execute task : ntpd_configure_do())
2022-02-12T19:57:39 Notice opnsense plugins_configure newwanip (execute task : dyndns_configure_do(,opt4))
2022-02-12T19:57:39 Notice opnsense plugins_configure newwanip (,opt4)
2022-02-12T19:57:39 Error opnsense /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN_Glasfaser.
2022-02-12T19:57:39 Notice opnsense plugins_configure vpn (execute task : openvpn_configure_do(,opt4))
2022-02-12T19:57:39 Notice opnsense plugins_configure vpn (execute task : ipsec_configure_do(,opt4))
2022-02-12T19:57:39 Notice opnsense plugins_configure vpn (,opt4)
2022-02-12T19:57:39 Error opnsense /usr/local/etc/rc.newwanip: IP address renew, killing all previous states
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: The WAN_GLASFASER_DHCP monitor address is empty, skipping.
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: The WAN_GLASFASER_DHCP6 monitor address is empty, skipping.
2022-02-12T19:57:38 Notice opnsense plugins_configure monitor (execute task : dpinger_configure_do())
2022-02-12T19:57:38 Notice opnsense plugins_configure monitor ()
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'fe80::ff:fe02:XXX%vtnet0'
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to fe80::ff:fe02:XXX
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to opt4
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '100.111.0.1'
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 100.111.0.1
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to opt4
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt4'
2022-02-12T19:57:38 Notice opnsense plugins_configure hosts (execute task : unbound_hosts_generate())
2022-02-12T19:57:38 Notice opnsense plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2022-02-12T19:57:38 Notice opnsense plugins_configure hosts ()
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: On (IP address: 100.111.133.XX) (interface: WAN_Glasfaser[opt4]) (real interface: vtnet0).
2022-02-12T19:57:38 Error opnsense /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'vtnet0'
2022-02-12T19:57:38 Notice dhclient Creating resolv.conf
2022-02-12T19:57:38 Notice dhclient route add default 100.111.0.1
2022-02-12T19:57:38 Notice dhclient New Routers (vtnet0): 100.111.0.1
2022-02-12T19:57:38 Notice dhclient New Broadcast Address (vtnet0): 100.111.255.255
2022-02-12T19:57:38 Notice dhclient New Subnet Mask (vtnet0): 255.255.0.0
2022-02-12T19:57:38 Notice dhclient New IP Address (vtnet0): 100.111.133.XX


Greets

Byte
February 16, 2022, 09:08:19 AM #1 Last Edit: February 16, 2022, 09:47:51 AM by Bytechanger
Now again?!
Is there an error on renew??

At about 08:58 the internet looses connection?!
So it seems that "Notice   configctl   event @ 1644997684.66 msg: Feb 16 08:48:04 OPNsense.localdomain config[16891]: [2022-02-16T08:48:04+01:00][info] config-event: new_config /conf/backup/config-1644997684.6553.xml" starts when it went wrong?!?!
I didn´t change anything or starts a backup job?! What mean new config?!
Is there a way to see more?
I think, every 24 hour it went down for 5 Minutes,...
there is an error


Code Select
2022-02-16T09:03:06 Notice opnsense plugins_configure newwanip (execute task : webgui_configure_do(,opt4))
2022-02-16T09:03:06 Notice opnsense plugins_configure newwanip (execute task : vxlan_configure_interface())
2022-02-16T09:03:06 Notice opnsense plugins_configure newwanip (execute task : unbound_configure_do(,opt4))
2022-02-16T09:03:06 Notice opnsense plugins_configure newwanip (execute task : openssh_configure_do(,opt4))
2022-02-16T09:03:06 Notice opnsense plugins_configure newwanip (execute task : opendns_configure_do())
2022-02-16T09:03:05 Notice opnsense plugins_configure newwanip (execute task : ntpd_configure_do())
2022-02-16T09:03:05 Notice opnsense plugins_configure newwanip (execute task : dyndns_configure_do(,opt4))
2022-02-16T09:03:05 Notice opnsense plugins_configure newwanip (,opt4)
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN_Glasfaser.
2022-02-16T09:03:05 Notice opnsense plugins_configure vpn (execute task : openvpn_configure_do(,opt4))
2022-02-16T09:03:05 Notice opnsense plugins_configure vpn (execute task : ipsec_configure_do(,opt4))
2022-02-16T09:03:05 Notice opnsense plugins_configure vpn (,opt4)
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: IP address renew, killing all previous states
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: The WAN_GLASFASER_DHCP monitor address is empty, skipping.
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: The WAN_GLASFASER_DHCP6 monitor address is empty, skipping.
2022-02-16T09:03:05 Notice opnsense plugins_configure monitor (execute task : dpinger_configure_do())
2022-02-16T09:03:05 Notice opnsense plugins_configure monitor ()
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'fe80::ff:XXX:XXX%vtnet0'
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to fe80::ff:XXXX:XXX
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to opt4
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '100.111.0.1'
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 100.111.0.1
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to opt4
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt4'
2022-02-16T09:03:05 Notice opnsense plugins_configure hosts (execute task : unbound_hosts_generate())
2022-02-16T09:03:05 Notice opnsense plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2022-02-16T09:03:05 Notice opnsense plugins_configure hosts ()
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: On (IP address: 100.111.133.XX) (interface: WAN_Glasfaser[opt4]) (real interface: vtnet0).
2022-02-16T09:03:05 Error opnsense /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'vtnet0'
2022-02-16T09:03:05 Notice dhclient Creating resolv.conf
2022-02-16T09:03:05 Notice dhclient route add default 100.111.0.1
2022-02-16T09:03:05 Notice dhclient New Routers (vtnet0): 100.111.0.1
2022-02-16T09:03:05 Notice dhclient New Broadcast Address (vtnet0): 100.111.255.255
2022-02-16T09:03:05 Notice dhclient New Subnet Mask (vtnet0): 255.255.0.0
2022-02-16T09:03:05 Notice dhclient New IP Address (vtnet0): 100.111.133.XX
2022-02-16T09:02:00 Error configctl error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2022-02-16T09:02:00 Error configctl error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2022-02-16T09:02:00 Error configctl error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out
2022-02-16T08:52:26 Notice configctl event @ 1644997945.97 exec: system event config_changed
2022-02-16T08:52:26 Notice configctl event @ 1644997945.97 msg: Feb 16 08:52:25 OPNsense.localdomain config[16891]: [2022-02-16T08:52:25+01:00][info] config-event: new_config /conf/backup/config-1644997945.9732.xml
2022-02-16T08:48:05 Notice configctl event @ 1644997684.66 exec: system event config_changed
2022-02-16T08:48:05 Notice configctl event @ 1644997684.66 msg: Feb 16 08:48:04 OPNsense.localdomain config[16891]: [2022-02-16T08:48:04+01:00][info] config-event: new_config /conf/backup/config-1644997684.6553.xml
February 16, 2022, 11:35:14 AM #2
How do you establish your WAN connection? PPPoE or DHCP?
How is the WAN interface configured?

If you are using PPPoE:
How does your PPP log look like? IPCP is interesting.

Is there a known issue by your upstream Provider regarding dialin?
i want all services to run with wirespeed and therefore run this dedicated hardware configuration:

AMD Ryzen 7 9700x
ASUS Pro B650M-CT-CSM
64GB DDR5 ECC (2x KSM56E46BD8KM-32HA)
Intel XL710-BM1
Intel i350-T4
2x SSD with ZFS mirror
PiKVM for remote maintenance

private user, no business use
February 21, 2022, 02:29:59 PM #3 Last Edit: February 21, 2022, 02:38:51 PM by Bytechanger
Hi,

its DHCP (Fibre Modem).

And again it´s off, and took a long time (Minutes) to restart internet...
And it´s the SAME IP, no change!

Code Select

2022-02-21T14:20:08 Error opnsense /usr/local/etc/rc.routing_configure: The WAN_GLASFASER_DHCP monitor address is empty, skipping.
2022-02-21T14:20:08 Error opnsense /usr/local/etc/rc.routing_configure: The WAN_GLASFASER_DHCP6 monitor address is empty, skipping.
2022-02-21T14:20:08 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: keeping current default gateway 'fe80::ff:fe02:xx%vtnet0'
2022-02-21T14:20:08 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: setting IPv6 default route to fe80::ff:fe02:xx
2022-02-21T14:20:08 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: IPv6 default gateway set to opt4
2022-02-21T14:20:08 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: keeping current default gateway '100.111.0.1'
2022-02-21T14:20:08 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: setting IPv4 default route to 100.111.0.1
2022-02-21T14:20:08 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: IPv4 default gateway set to opt4
2022-02-21T14:20:08 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
2022-02-21T14:19:54 Error opnsense /usr/local/etc/rc.routing_configure: The WAN_GLASFASER_DHCP monitor address is empty, skipping.
2022-02-21T14:19:54 Error opnsense /usr/local/etc/rc.routing_configure: The WAN_GLASFASER_DHCP6 monitor address is empty, skipping.
2022-02-21T14:19:54 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: keeping current default gateway 'fe80::ff:fe02:xx%vtnet0'
2022-02-21T14:19:54 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: setting IPv6 default route to fe80::ff:fe02:202
2022-02-21T14:19:54 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: IPv6 default gateway set to opt4
2022-02-21T14:19:54 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: keeping current default gateway '100.111.0.1'
2022-02-21T14:19:54 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: setting IPv4 default route to 100.111.0.1
2022-02-21T14:19:54 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: IPv4 default gateway set to opt4
2022-02-21T14:19:54 Error opnsense /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
2022-02-21T14:18:23 Error opnsense /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN_Glasfaser.
2022-02-21T14:18:23 Error opnsense /usr/local/etc/rc.newwanip: IP address renew, killing all previous states
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: The WAN_GLASFASER_DHCP monitor address is empty, skipping.
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: The WAN_GLASFASER_DHCP6 monitor address is empty, skipping.
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'fe80::ff:fe02:xxx%vtnet0'
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to fe80::ff:fe02:xxx
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to opt4
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '100.111.0.1'
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 100.111.0.1
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to opt4
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt4'
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: On (IP address: 100.111.133.xx) (interface: WAN_Glasfaser[opt4]) (real interface: vtnet0).
2022-02-21T14:18:22 Error opnsense /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'vtnet0'
February 22, 2022, 06:20:36 AM #4
February 22, 2022, 11:18:58 AM #5
Looks like my problem. Do you have a easy setup for your wireguard restart script until this is fixed?
February 22, 2022, 11:27:52 AM #6 Last Edit: February 22, 2022, 11:29:53 AM by Bytechanger
I use monit to check wireguard connection and if ping fails 5 times the adapter is restart.

I´m not sure if wireguard is the problem?!


So I use monit to check connection, if down i use

/usr/local/bin/wg-quick down wg1
/usr/local/bin/wg-quick up wg1

See Example 2 https://docs.opnsense.org/manual/monit.html

To restart service.
Works fine.
March 02, 2022, 08:55:17 AM #7
The problem still persists in 22.1.2. WAN comes down quite often now when I have a cloud sync task running, using the upstream of my connection for hours to transfer hundreds of gigabytes to the backup service.

I've had the connection dropping now every hour or so. What I see is there's lots of activity in pf.searches value during the upload. Restarting the Wireguard connection makes the connection to work again.
Print
Go Up Pages1
User actions