Multi-WAN not routing incoming packets back whence they came properly

[drivera]

Hi!

I have a Multi-WAN setup, which after some toil (mostly due to my newbness ) appears to be (mostly) working the way I want it to (thanks to mimugmail for helping me out!). However, there's one thing not working right now that I can't see my way past.

The WANs are set up for a failover scenario: if the primary fails, the secondary takes over. This works well enough. The problem is that while everything is up (i.e. primary is up), I'm unable to ping the secondary interface from a remote location. Pinging the primary works just fine.  When the primary is down and the secondary is up, then I can ping the secondary (now primary due to the failover) just fine from that same location.

The issue, I believe, has to do with default gateways. I can only set up one default gateway. I had to enable gateway switching to get around other problems (discussed here, there's some more fun hijinks on that topic but I digress).


Using the Packet Capture utility I can see that the traffic does arrive fine to the firewall on the secondary while the primary is up.  The problem is that a response is never sent out. This is because the primary had to be set as the default gateway (see the above link) for gateway switching to work, so the O/S (apparently) doesn't know to give those packets special treatment and bounce them right back the network interface they came from.


I know OPNSense isn't Linux, but the way to solve this in Linuxland would be to have a routing rule (using ip route) specifying that packets originating from a given interface's address are to be routed using a special routing table (built for that interface) where the default gateway is that interface's.


I have no clue how to do that on OPNSense-land (*BSD-land)...


Can you guys help me out?

[jf5876]

Go into each interface, and at the bottom where you can select a gateway, select the correct gateway.

Also, check out Firewall -> Settings -> Advanced and check Sticky Connections under Multi-WAN :-D

[drivera]

Sticky connections was already enabled, and per-interface gateway selection appears to only be possible for statically-configured interfaces (none of mine are - all are DHCP).

Any other ideas?

[mimugmail]

Screenshot of Firewall : Settings : Advanced please ...

[drivera]

Here you go... anything else I can provide you to help diagnose?

[mimugmail]

Kill states and Disable Force Gateway is enabled on my side.

[drivera]

Enabling either (or both) setting(s) had no effect. Even rebooting once they were enabled. Any other thoughts?

[mimugmail]

I'll check tomorrow, have a customer with similar setup, also with DNAT for both WAN IPs to same host, so it must ne working anyhow

[drivera]

I'm sure it's some little detail somewhere. I can send you a (sanitized) copy of my configuration, if that would help.

Cheers!