Dual WAN: WAN2 fine, WAN1 doesn't route

[guest36697]

The Setup

WAN1: PPPoE (both IPv4 and IPv6)

WAN2: Starlink (in bridged mode)

WAN_GROUP: WAN1 as Tier 1, WAN2 as Tier 2

Probably not relevant, but here just in case:

LAN: vlan0 (device group A), vlan1 (device group B), vlan2 (IOT devices)

There's a wireguard gateway that all vlan0 devices route over (vlan1 -> wg_WAN), everyone else (that need internet access) route over WAN2 directly (outbound NAT rule: vlan2 -> WAN2).

The Situation

Everything was working fine. WAN1 was the primary WAN, wg_WAN connected over WAN1, and all client devices routed fine (all on IPV4). We moved the Starlink to a better location since trees were causing some interference issues, but clients on vlan2 were routing fine over WAN2 before the move.

Disconnected WAN2 and proceeded to move the Starlink. Decided that some of the cable routing needed to be improved, removed the WAN1, adjusted the run, and reconnected. Noticed in the web interface that the PPPoE connection hadn't come back up - not a problem, sometimes it gets caught in a LCP reconnection attempt loop - fixed that, but the WAN1 gateway kept coming up as down even though it was getting its static IP from the ISP.

Routing table had no default route. Had the routes to the DNS and to the CGNAT gateway, but that was about it.

Plugged the WAN2 cable once all was set up again with Starlink, immediately created the default route, and the WAN_GROUP did the job (routes over WAN2, continues to mark WAN1 as down).

I checked the cable, swapped it; thought it could be the ONT (rebooted, got reconnected, got the expected IP); enabled IPV6 to make sure that we were getting a good connection, and yes, IPv6 works - gateway marked as up and does see traffic. Rebooted thinking that something may have gotten into a stale state. Updated to 23.7.11 (was running 23.7.10_1 - which was working before the change).

I have no idea what's going on - if I disconnect WAN2, it doesn't failover (WAN1 is seen as 100% packet loss) and it doesn't recreate the default route to WAN1. I can ping the public IP from inside the network, but if I run a traceroute or a ping on the firewall, it fails. I can see the interface sending the ping to the gateway monitor, the firewall is letting the traffic through, but not getting a response (used 1.1.1.1 , 8.8.8.8, and another known good IP).

TL;DR: Multi-WAN config, WAN2 works fine, WAN1 won't route. Default route will be created on WAN1.

Any ideas where I should poke next?

[knebb]

Do you have a rule for your lan to use the defined gateway groups?

[guest36697]

The LAN has no problem in routing - no rules have changed between disconnecting the WAN1/WAN2 (for the relocation and recabling), but these are the rules as they stand today (in order):

NAT > Outbound
- Let vlan0 access the management ip of the Starlink bridge device
- Let vlan0 route over the Wireguard interface
- Let vlan1 route over the bare Starlink interface

Rules > LAN
- If vlan0 network and destination is !RFC1918_Networks, let vlan0 route over wireguard gateway
- Default all any LAN to any gateway rule

Neither the WAN1 or WAN2 interfaces have any additional rules.

Nothing fancy in the rules, and these were working before (and survived several reboots), I would be really keen on learning why they were working before but not anymore.

[knebb]

I do not see you vlan* have a route set to the WAN_GROUP.

You have configured this group but you do not use it.

But for further troubleshooting, make sure both WAN connections are really fine up and running. As long as this is not true, any other will not work properly.
Try to ping the received default gateways for each WAN interface from you OPNSense and make sure with traceroute or packet capturing if they leave on the correct WAN interface. Repeat, if needed for IPv6.

Once this is fine, wen can go further and make sure your vlans route through the defined interfaces.

Working here like a champ. But there is a little bit mor work to be done for Multi-WAN. Are you sure you did all documented things?

[guest36697]

Hey knebb, yeah - if you're asking about vlan2, it does have a rule to route over the WAN2 gateway, and that works perfectly. The vlans are routing according to their rules (either over wireguard if a member of vlan1 or over the bare WAN2 gateway if a member of vlan2).

The Multi-WAN part *should* work (it was working before - if WAN1 failed, wireguard would reconnect on WAN2) if we can figure out why WAN1 is being reported as down (100% packet loss).

There are 2 behaviors that I can be sure of right now:
- if I disconnect WAN2, leaving WAN1 connected, it will not create the default route for WAN1
- dpinger for WAN1 reports WAN1 as down (100% packet loss) with a few sendto error: 50 and 64 in the logs yesterday, but not since (restarting dpinger for WAN1 does not change this).

WAN1 does get the expected ip from the ISP and does get an ipv6 using ipv4 connectivity. The interface is reported as up, and there is traffic flowing over the ipv6 gateway. Packet capture on ipv6 confirms this. Packet capture on ipv4 shows dpinger trying to get a response but never seeing one come back.