Multi-WAN with transit network, servers and LAN

[OPNsense4ever]

Hello! I have two different WAN connections:

Primary WAN: Connected via a /30 transit network to a /29 network, similar to Comcast EDI. One IP from the /29 is assigned to an OPT interface on the main OPNsense router. Other routers behind this interface use that IP as their default gateway.

Backup WAN: Provides a DHCP-assigned public IP.

Goal:

LAN Traffic: I need the LAN connection behind these two WAN connections to be as bulletproof as possible. During work hours, I can't afford any latency or packet loss. Therefore, I want the LAN to fail over to the backup WAN immediately when there's any issue with the primary WAN.

OPT Interface Traffic: I want the public IPs in the /29 network (used by the OPT interface) to stay up as much as possible, even if there's some latency or packet loss on the primary connection. Essentially, I prefer that the gateways remain marked as up for the OPT interface, even when the primary WAN has minor issues.

Issue:

When I experience packet loss or latency on the primary network and OPNsense switches to the backup WAN for failover, the routers using the OPT interface's IP lose their connection completely until the primary WAN recovers. I believe this happens because OPNsense tries to route the /29 network traffic through the backup WAN, which doesn't support it.

Question:

Is there a way to configure OPNsense so that:

LAN Traffic: Fails over to the backup WAN when there's latency or packet loss on the primary WAN.

OPT Interface Traffic: Continues to use the primary WAN (via the /30 transit network) exclusively, regardless of the gateway's status, unless the primary WAN is completely down.

Current Configuration:

I've set up the EDI-like network similar to this guide: https://meh.roach.xxx/2024/04/26/comcast-edi-with-opnsense-route-public-ips-through-opnsense/

Summary:

I need the LAN to fail over to the backup WAN immediately during any latency or packet loss on the primary WAN to maintain reliable connectivity for work-related applications (like Zoom and VPNs).

I want the OPT interface (and the public IPs in the /29 network) to continue using the primary WAN even during minor issues, to maintain services that rely on those public IPs.

I'm looking for the best way to configure OPNsense to accommodate these requirements without adding another physical router.

Any advice or guidance would be greatly appreciated!

[dseven]

I think you should be able to do that by specifying the gateway (your "primary WAN" one) in firewall rules for your OPT interface (your "allow OPT to any" rule, and anything else for WAN destinations).

[OPNsense4ever]

I would think that too, but it looks like as soon as OPNsense marks the gateway of the primary WAN down due to quality issues (like 10% packet loss) then traffic just stops getting routed to it for both the NAT on the LAN and the public IPs.

Is there another switch or tick somewhere I should be looking for?