Multi-WAN router-originated traffic

[obrienmd]

I'll be configuring something in our test lab this week to debug further, and will report back!

[whitwye]

Thanks. Tried one more thing: I'd had the WAN2 set to take over outward routing as failover. Reconfigured for it to instead work in load balancing mode. Didn't make a difference. Turning off the first WAN interface results in traffic not being responded to when sent to WAN2 IPs (that had been working with WAN1 on in either case), nor in the Firewall being able to initiate any outgoing traffic.

I can see interesting changes with

diff rules.debug rules.debug.old

in /tmp, with route-to and reply-to rules changing with the configuration changes and turning WAN1 off and on. So the system's not failing to recognize the changes. It's just not responding with full adequacy.

If anyone has additional configuration steps to suggest which might work around this, I'm up for more experimentation.

[whitwye]

Note: Tried the patch here: https://forum.opnsense.org/index.php?topic=5785.0 (and above in this thread). As I noted there, it does not fix what I'm seeing.

[whitwye]

Another data point: Disabling WAN2 has no effect on WAN1.

So:

Disable WAN1 and WAN2 no longer can respond to outside traffic coming in, nor originate traffic. (There's nothing yet using this system for LAN devices going outwards, so haven't tested that.)

Disable WAN2 and WAN1 continues working for both outside traffic coming in, and originating traffic.

Checking with "netstat -nr" disabling WAN1 removes the default route via WAN1, and does not replace it with a default route via WAN2. WAN2 does have its IPv4 Upstream Gateway set in the configuration, but that is not substituted in this case.

[mimugmail]

I'm experienceing the same right now!

https://forum.opnsense.org/index.php?topic=5942.0
https://github.com/opnsense/core/issues/1811
https://github.com/opnsense/core/commit/0b38eff5f#commitcomment-24246290

Multi WAN with local services is ATM a b*tch

I'm in IRC within workhours, would like to troubleshoot with you guys together ...

[obrienmd]

Default gateway switching seems to be working OK right now. The local services (DNS, Zerotier, etc.) angle makes this the only real workable option for me ATM.

Franco's earlier notes are making much more sense to me - I know one of the big pains for people moving from Sonicwall / Fortinet / Watchguard boxes to OPNSense / pfSense / etc. is "multi-WAN is hard". Providing a guided UI to simplify multi-WAN would help quite a bit in these scenarios.

Peplink is one vendor that, while their boxes are pretty simple and don't do much, do the multi-WAN UI simplification (and heck, Multi-WAN itself) fairly well. Franco, I'd be happy to provide someone from OPNSense access if you'd like to peek and don't have any around.

[franco]

Hi Michael,

That would be great to have a look at if possible.

I would say the whole thing was "organically grown" adding features here and there through several pfSense / FreeBSD versions in the past. Everything makes good sense in the context of single iterations, but in the grand scheme of things where we are looking at it now there is potential for a revamp of how it is presented / configured.


Cheers,
Franco

[obrienmd]

Ah, turns out they have an online demo. Use admin/admin at:

https://balancedemo.peplink.com/cgi-bin/MANGA/index.cgi

It's not the prettiest UI, but the multi-WAN stuff works really well. Take a look at outbound policies in particular.

Generally, I think if you could tag a gateway group as "default gateway" for a box, that covered both internal and client traffic, that would be a great user experience - given the power gateway groups already have with parameters for "health", load balancing vs. failover, etc.

[obrienmd]

@franco - Perhaps worth moving into a request for feedback GitHub issue on "making multiWAN awesome"?