[SOLVED] Uplink randomly going down since OpnSense 18.7.7

[Vashiru]

Note that this topic is a copy of my Github ticket #2988, but I hope to reach a broader audience that might be able to help me resolve my issues. It drops out on average once every 2 days and it's becoming an unworkable situation.

About a 1,5 week ago I upgraded our OPNSense firewall from 18.7.5 to 18.7.7. This caused some minor issues where it seemed to have lost it's sense of DHCP and the uplink. A reboot and some tweaking later, it seemed to have been resolved.

Fast forward a few days and our internet suddenly drops out. After investigating the firewall has lost it's uplink. When I try to ping a host from the VGA console it tells me no route to host. When I tell it to reload all services (option 11 on the menu), it starts to work again. Sometimes for a few hours, sometimes for a few days.

I've checked the firewall logs for any errors or general stuff out of the ordinary and I noticed that on each moment of downtime /usr/local/etc/rc.newwanip has been run. Though I don't see anything being off looking at the messages from it (it gives the same ones upon Reload all), but I might be looking at the wrong place.

Our office has 2 external IP adresses, one starting with 84.x.x.x and another starting with 178.x.x.x, since it breaks on /usr/local/etc/rc.newwanip I think that might be relevant. Also we don't use IPv6.

In an attempt to resolve this I've upgraded it to 18.7.8, hoping that this would resolve the issue, but unfortunately it's still present. Is there maybe some setting I need to change between those versions in order to get this to work reliably or have I stumbled upon a bug somewhere?

Please let me know if you need any more information.

Firewall system.log:

Code: [Select]

Nov 29 18:31:54 OPNsense opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'bce0_vlan128'
Nov 29 18:31:54 OPNsense opnsense: /usr/local/etc/rc.newwanip: On (IP address: 84.x.x.x) (interface: WAN[wan]) (real interface: bce0_vlan128).
Nov 29 18:31:54 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Nov 29 18:31:54 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to lan
Nov 29 18:31:54 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Nov 29 18:31:54 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Nov 29 18:31:54 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Nov 29 18:31:56 OPNsense opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Nov 29 18:35:31 OPNsense kernel: ugen1.4: <Lite-On Tech USB 1.1 2 port downstream low-power hub> at usbus1
Nov 29 18:35:31 OPNsense kernel: uhub5 on uhub4
Nov 29 18:35:31 OPNsense kernel: uhub5: <Lite-On Tech USB 1.1 2 port downstream low-power hub, class 9/0, rev 1.10/2.01, addr 4> on usbus1
Nov 29 18:35:31 OPNsense kernel: uhub5: 4 ports with 2 removable, bus powered
Nov 29 18:35:32 OPNsense kernel: ugen1.5: <Lite-On Tech IBM USB Travel Keyboard with UltraNav> at usbus1
Nov 29 18:35:32 OPNsense kernel: ukbd0 on uhub5
Nov 29 18:35:32 OPNsense kernel: ukbd0: <Lite-On Tech IBM USB Travel Keyboard with UltraNav, class 0/0, rev 1.10/1.16, addr 5> on usbus1
Nov 29 18:35:32 OPNsense kernel: kbd0 at ukbd0
Nov 29 18:35:32 OPNsense kernel: uhid0 on uhub5
Nov 29 18:35:32 OPNsense kernel: uhid0: <Lite-On Tech IBM USB Travel Keyboard with UltraNav, class 0/0, rev 1.10/1.16, addr 5> on usbus1
Nov 29 18:35:33 OPNsense kernel: ugen1.6: <Synaptics Inc. Composite TouchPad  TrackPoint> at usbus1
Nov 29 18:35:33 OPNsense kernel: ums0 on uhub5
Nov 29 18:35:33 OPNsense kernel: ums0: <Rel> on usbus1
Nov 29 18:35:33 OPNsense kernel: ums0: 3 buttons and [XY] coordinates ID=0
Nov 29 18:35:33 OPNsense kernel: ums1 on uhub5
Nov 29 18:35:33 OPNsense kernel: ums1: <Rel> on usbus1
Nov 29 18:35:33 OPNsense kernel: ums1: 3 buttons and [XY] coordinates ID=0
Nov 29 18:44:43 OPNsense opnsense: user 'root' authenticated successfully
Nov 29 18:44:43 OPNsense login: login on ttyv0 as root
Nov 29 18:44:43 OPNsense login: ROOT LOGIN (root) ON ttyv0
Nov 29 18:44:43 OPNsense login: ROOT LOGIN (root) ON ttyv0
Nov 29 18:45:01 OPNsense kernel: ifa_maintain_loopback_route: deletion failed for interface bce0_vlan128: 3
Nov 29 18:45:01 OPNsense kernel: ifa_maintain_loopback_route: deletion failed for interface bce0_vlan128: 3
Nov 29 18:45:01 OPNsense kernel: vlan0: changing name to 'bce0_vlan128'
Nov 29 18:45:01 OPNsense kernel: ifa_maintain_loopback_route: deletion failed for interface bce1: 3
Nov 29 18:45:03 OPNsense opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'bce0_vlan128'
Nov 29 18:45:04 OPNsense opnsense: /usr/local/etc/rc.newwanip: On (IP address: 84.x.x.x) (interface: WAN[wan]) (real interface: bce0_vlan128).
Nov 29 18:45:04 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Nov 29 18:45:04 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to lan
Nov 29 18:45:04 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Nov 29 18:45:04 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Nov 29 18:45:04 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Nov 29 18:45:05 OPNsense opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Nov 29 18:45:05 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: entering configure using defaults
Nov 29 18:45:05 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: IPv4 default gateway set to lan
Nov 29 18:45:05 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: no IPv6 default gateway set, assuming wan
Nov 29 18:45:05 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: setting IPv4 default route to 84.x.x.x
Nov 29 18:45:05 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: removing /tmp/bce1_defaultgw
Nov 29 18:45:05 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: creating /tmp/bce1_defaultgw using '84.x.x.x'
Nov 29 18:45:05 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: skipping IPv6 default route
Nov 29 18:45:09 OPNsense opnsense: /usr/local/etc/rc.reload_all: Resyncing OpenVPN instances.
Nov 29 18:45:09 OPNsense kernel: ifa_maintain_loopback_route: deletion failed for interface ovpns1: 3
Nov 29 18:45:09 OPNsense kernel: ovpns1: link state changed to DOWN
Nov 29 18:45:11 OPNsense kernel: ovpns1: link state changed to UP
Nov 29 18:45:11 OPNsense opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ovpns1'
Nov 29 18:45:11 OPNsense opnsense: /usr/local/etc/rc.newwanip: Interface '' is disabled or empty, nothing to do.
Nov 29 18:47:38 OPNsense opnsense: /index.php: Session timed out for user 'root' from: 192.168.1.243
Nov 29 18:47:38 OPNsense api[4891]: no active session, user not found
Nov 29 18:47:48 OPNsense opnsense: /index.php: Successful login for user 'root' from: 192.168.1.243
Nov 29 19:11:30 OPNsense sshlockout[1676]: sshlockout/webConfigurator v3.0 starting up
Nov 29 20:13:41 OPNsense kernel: ugen1.4: <Lite-On Tech USB 1.1 2 port downstream low-power hub> at usbus1 (disconnected)
Nov 29 20:13:41 OPNsense kernel: uhub5: at uhub4, port 1, addr 4 (disconnected)
Nov 29 20:13:41 OPNsense kernel: ugen1.5: <Lite-On Tech IBM USB Travel Keyboard with UltraNav> at usbus1 (disconnected)
Nov 29 20:13:41 OPNsense kernel: ukbd0: at uhub5, port 3, addr 5 (disconnected)
Nov 29 20:13:41 OPNsense kernel: ukbd0: detached
Nov 29 20:13:41 OPNsense kernel: uhid0: at uhub5, port 3, addr 5 (disconnected)
Nov 29 20:13:41 OPNsense kernel: uhid0: detached
Nov 29 20:13:41 OPNsense kernel: ugen1.6: <Synaptics Inc. Composite TouchPad  TrackPoint> at usbus1 (disconnected)
Nov 29 20:13:41 OPNsense kernel: ums0: at uhub5, port 4, addr 6 (disconnected)
Nov 29 20:13:41 OPNsense kernel: ums0: detached
Nov 29 20:13:41 OPNsense kernel: ums1: at uhub5, port 4, addr 6 (disconnected)
Nov 29 20:13:41 OPNsense kernel: ums1: detached
Nov 29 20:13:41 OPNsense kernel: uhub5: detached
Nov 29 21:35:12 OPNsense opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'bce0_vlan128'
Nov 29 21:35:12 OPNsense opnsense: /usr/local/etc/rc.newwanip: On (IP address: 84.x.x.x) (interface: WAN[wan]) (real interface: bce0_vlan128).
Nov 29 21:35:13 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Nov 29 21:35:13 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to lan
Nov 29 21:35:13 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Nov 29 21:35:13 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Nov 29 21:35:13 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Nov 29 21:35:14 OPNsense opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Nov 30 09:52:52 OPNsense kernel: ugen1.4: <Lite-On Tech USB 1.1 2 port downstream low-power hub> at usbus1
Nov 30 09:52:52 OPNsense kernel: uhub5 on uhub4
Nov 30 09:52:52 OPNsense kernel: uhub5: <Lite-On Tech USB 1.1 2 port downstream low-power hub, class 9/0, rev 1.10/2.01, addr 4> on usbus1
Nov 30 09:52:53 OPNsense kernel: uhub5: 4 ports with 2 removable, bus powered
Nov 30 09:52:54 OPNsense kernel: ugen1.5: <Lite-On Tech IBM USB Travel Keyboard with UltraNav> at usbus1
Nov 30 09:52:54 OPNsense kernel: ukbd0 on uhub5
Nov 30 09:52:54 OPNsense kernel: ukbd0: <Lite-On Tech IBM USB Travel Keyboard with UltraNav, class 0/0, rev 1.10/1.16, addr 5> on usbus1
Nov 30 09:52:54 OPNsense kernel: kbd0 at ukbd0
Nov 30 09:52:54 OPNsense kernel: uhid0 on uhub5
Nov 30 09:52:54 OPNsense kernel: uhid0: <Lite-On Tech IBM USB Travel Keyboard with UltraNav, class 0/0, rev 1.10/1.16, addr 5> on usbus1
Nov 30 09:52:54 OPNsense kernel: ugen1.6: <Synaptics Inc. Composite TouchPad  TrackPoint> at usbus1
Nov 30 09:52:54 OPNsense kernel: ums0 on uhub5
Nov 30 09:52:54 OPNsense kernel: ums0: <Rel> on usbus1
Nov 30 09:52:54 OPNsense kernel: ums0: 3 buttons and [XY] coordinates ID=0
Nov 30 09:52:55 OPNsense kernel: ums1 on uhub5
Nov 30 09:52:55 OPNsense kernel: ums1: <Rel> on usbus1
Nov 30 09:52:55 OPNsense kernel: ums1: 3 buttons and [XY] coordinates ID=0
Nov 30 09:53:44 OPNsense opnsense: user 'root' authenticated successfully
Nov 30 09:53:44 OPNsense login: login on ttyv0 as root
Nov 30 09:53:44 OPNsense login: ROOT LOGIN (root) ON ttyv0
Nov 30 09:53:44 OPNsense login: ROOT LOGIN (root) ON ttyv0
Nov 30 09:53:47 OPNsense kernel: ifa_maintain_loopback_route: deletion failed for interface bce0_vlan128: 3
Nov 30 09:53:47 OPNsense kernel: ifa_maintain_loopback_route: deletion failed for interface bce0_vlan128: 3
Nov 30 09:53:47 OPNsense kernel: vlan0: changing name to 'bce0_vlan128'
Nov 30 09:53:47 OPNsense kernel: ifa_maintain_loopback_route: deletion failed for interface bce1: 3
Nov 30 09:53:50 OPNsense opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'bce0_vlan128'
Nov 30 09:53:50 OPNsense opnsense: /usr/local/etc/rc.newwanip: On (IP address: 84.x.x.x) (interface: WAN[wan]) (real interface: bce0_vlan128).
Nov 30 09:53:50 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Nov 30 09:53:50 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to lan
Nov 30 09:53:50 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Nov 30 09:53:50 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Nov 30 09:53:50 OPNsense opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Nov 30 09:53:51 OPNsense opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Nov 30 09:53:51 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: entering configure using defaults
Nov 30 09:53:51 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: IPv4 default gateway set to lan
Nov 30 09:53:51 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: no IPv6 default gateway set, assuming wan
Nov 30 09:53:51 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: setting IPv4 default route to 84.x.x.x
Nov 30 09:53:51 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: removing /tmp/bce1_defaultgw
Nov 30 09:53:51 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: creating /tmp/bce1_defaultgw using '84.x.x.x'
Nov 30 09:53:51 OPNsense opnsense: /usr/local/etc/rc.reload_all: ROUTING: skipping IPv6 default route
Nov 30 09:53:56 OPNsense opnsense: /usr/local/etc/rc.reload_all: Resyncing OpenVPN instances.
Nov 30 09:53:56 OPNsense kernel: ifa_maintain_loopback_route: deletion failed for interface ovpns1: 3
Nov 30 09:53:56 OPNsense kernel: ovpns1: link state changed to DOWN
Nov 30 09:53:57 OPNsense kernel: ovpns1: link state changed to UP
Nov 30 09:53:57 OPNsense opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ovpns1'
Nov 30 09:53:57 OPNsense opnsense: /usr/local/etc/rc.newwanip: Interface '' is disabled or empty, nothing to do.
Nov 30 10:02:37 OPNsense kernel: ugen1.4: <Lite-On Tech USB 1.1 2 port downstream low-power hub> at usbus1 (disconnected)
Nov 30 10:02:37 OPNsense kernel: uhub5: at uhub4, port 1, addr 4 (disconnected)
Nov 30 10:02:37 OPNsense kernel: ugen1.5: <Lite-On Tech IBM USB Travel Keyboard with UltraNav> at usbus1 (disconnected)
Nov 30 10:02:37 OPNsense kernel: ukbd0: at uhub5, port 3, addr 5 (disconnected)
Nov 30 10:02:37 OPNsense kernel: ukbd0: detached
Nov 30 10:02:37 OPNsense kernel: uhid0: at uhub5, port 3, addr 5 (disconnected)
Nov 30 10:02:37 OPNsense kernel: uhid0: detached
Nov 30 10:02:37 OPNsense kernel: ugen1.6: <Synaptics Inc. Composite TouchPad  TrackPoint> at usbus1 (disconnected)
Nov 30 10:02:37 OPNsense kernel: ums0: at uhub5, port 4, addr 6 (disconnected)
Nov 30 10:02:37 OPNsense kernel: ums0: detached
Nov 30 10:02:37 OPNsense kernel: ums1: at uhub5, port 4, addr 6 (disconnected)
Nov 30 10:02:37 OPNsense kernel: ums1: detached
Nov 30 10:02:37 OPNsense kernel: uhub5: detached

Relevant interface:

Code: [Select]

bce0_vlan128: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether bc:30:5b:d7:01:f4
        inet6 fe80::be30:5bff:fed7:1f4%bce0_vlan128 prefixlen 64 scopeid 0x8
        inet 84.x.x.x netmask 0xffffff00 broadcast 84.x.x.x
        inet 178.x.x.x netmask 0xffffffff broadcast 178.x.x.x
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex,master>)
        status: active
        vlan: 128 vlanpcp: 0 parent interface: bce0
        groups: vlan

I had a look at the ipconfig before and after doing a 'Reload all services' but nothing changed there.

[franco]

What's in front of the OPNsense? ISP modems/routers?


Cheers,
Franco

[Vashiru]

What's in front of the OPNsense? ISP modems/routers?


Cheers,
Franco

Just an ISP modem.

[Vashiru]

Had a chance to take a peak at which modem it is specifically. It appears to be a Genexis FiberXport OCG-1012. We have a fiber connection.

I'm assuming the info I've given in the start post is too little, what other information might be useful to add?

[Vashiru]

So I think I may have resolved the issue. I have marked one of my gateways as the default gateway and the logging now looks like this:

Code: [Select]

Dec 19 19:49:04 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'bce0_vlan128'
Dec 19 19:49:04 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 84.x.x.x) (interface: WAN[wan]) (real interface: bce0_vlan128).
Dec 19 19:49:04 opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Dec 19 19:49:04 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Dec 19 19:49:04 opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Dec 19 19:49:04 opnsense: /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 84.x.x.y
Dec 19 19:49:04 opnsense: /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '84.x.x.y'
Dec 19 19:49:04 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Dec 19 19:49:05 opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN.

So it actually sets the IPv4 gateway to WAN instead of LAN now and doesn't delete any routes that would be outgoing. I'll keep an eye on it for a few days and if it's resolved I'll update this post.

EDIT 07-01-2019: Setting the WAN interface as default resolved the issue.