Multi-WAN Bug/Oversight?

[db]

Background:

I have a multi-wan setup with multiple ISPs, for both fault tolerance as well as to increase available bandwidth on my network.

I recently purchased a second service from one of those providers, and added it as another WAN. Both of these are Tier 1 in a gateway group, and most of my traffic is directed through this gateway group as a load balanced group with sticky connections.

I've only had this set up for a few days, and for the most part, it seems to be working (so far as load balancing goes), but there are some.. oddities, which I believe are due to both WAN gateways having the same gateway IP (due to them being from the same ISP).

Relevant Info:

WAN4 is cgnat and has a gateway address of 100.64.0.1

WAN5 is cgnat and has a gateway address of 100.64.0.1

I manually added monitor ips for each, because otherwise having them both use 100.64.0.1 seemed like it would really only be monitoring one of them (because of routing).

WAN4 has monitor IP of 1.0.0.1

WAN5 has monitor IP of 1.1.1.1

WAN4 is on interface IGB4

WAN5 is on interface IGB5

I have DNS servers assigned to each:

WAN4 should be 208.67.220.220

WAN5 should be 208.67.222.222

Both also have IPs assigned from the ISP:

WAN4 has IP 100.68.*.*

WAN5 has IP 100.120.*.*

Issues:

Mostly, routing seems to be based on gateway IP and not interface, so I have:

default, 100.64.0.1, UGS, igb4

1.0.0.1, 100.64.0.1, UGHS, igb4

(nothing for 1.1.1.1)

208.67.220.220, 100.64.0.1, UGHS, igb4

208.67.222.222, 100.64.0.1, UGHS, igb4

Notice the issues? Monitor IP for WAN5/IGB5 of 1.1.1.1 will be routed via the default gateway, so will actually be monitoring WAN4/IGB4. Also DNS will only be using WAN4/IGB4.

I can manually edit the route table (using 'route') and make it all make sense, but something overwrites my changes after a minute or so.

Is this a bug or oversight? Am I doing something odd having two connections from the same ISP (and thus the same gateway IP)? It doesn't seem that odd.

Am I doing something wrong, or should I file this as a bug?

[franco]

FreeBSD never supported same gateway on two different interfaces. There have been many similar forum questions in the past.


Cheers,
Franco

[db]

That's unfortunate, I'm not in control of the gateway IPs. The gateway group using both as Tier 1 does seem to 'work' however (I'm seen coming from both IPs).

I'm a bit confused when you say this isn't supported in FreeBSD, does this not do what I think it's doing?

route get 208.67.222.222
route to: dns.opendns.com
destination: dns.opendns.com
gateway: 100.64.0.1
interface: igb4

route change -net 208.67.222.222 -interface igb5

route get 208.67.222.222
route to: dns.opendns.com
destination: dns.umbrella.com
interface: igb5

[franco]

I think default gateway switching works, but if you pin a route through the gateway and both are attached the second interface is always ignored so you can't use them at the same time/monitor them independently.


Cheers,
Franco

[db]

I suppose a sledgehammer option would be to run another opnsense box in front of at least one of these, and have it NAT through 100.64.0.1 to something else so the first opnsense box sees a different IP... unless there might be some way to do that with virtual nics on one opnsense box but I can't come up with a way how.

I am sort of confused how the load balancing is working right now. I'd say maybe the opnsense stats are lying to me but if I navigate to any 'what is my ip' site I will see both external IPs.

[lilsense]

I am not sure what your ISP is, but no one in the right state of mind would provide same IP on two external connections.

Also, on another note the ONLY time two interfaces can have the same default gateway, it would be on a switch not a firewall/router/PC.