Multi WAN - what am I missing here ?

[reep]

Scratching my head here.

Setup Multi WAN with 2 x WAN connections as per the docs.

What I can't figure out is what to do with setting (or not) the default gateway on a WAN.

Clearly you can only set a default route on one connection or the other. But I presume (though it is not mentioned) that this should be disabled for both WAN connections in a Multi WAN setup.

OK, so no default gateway and we add a rule to forward all local traffic to the gateway group.

From Opnsense I can ping the gateway monitor IPs and they traceroute out via each WAN device correctly.

From Opnsense I can ping the DNS servers set by the WAN routers and they traceroute out via each WAN device correctly.

For the life of me what I cannot do is ping any other host. I just get "No route to host"

That kind of makes sense - there are routes set for the DNS IPs and for the gateway monitor IPs but no other traffic so the packets should be picked up by the firewall rule but nothing happens.

Nothing appears in the firewall logs.

You can see the routes set for the Googler DNS servers used as gateway monitor IPs and you can see the DNS servers set by the ADSL routers

Internet:
Destination        Gateway            Flags      Netif Expire
8.8.4.4            192.168.2.1        UGHS     vtnet3
8.8.8.8            192.168.1.1        UGHS     vtnet1
10.0.0.0/24        link#1             U        vtnet0
10.0.0.251         link#1             UHS         lo0
127.0.0.1          link#8             UH          lo0
192.168.1.0/24     link#2             U        vtnet1
192.168.1.11       link#2             UHS         lo0
192.168.2.0/24     link#4             U        vtnet3
192.168.2.11       link#4             UHS         lo0
208.67.220.220     192.168.2.1        UGHS     vtnet3
208.67.222.222     192.168.1.1        UGHS     vtnet1

A traceroute to any of those 4 IPs shows the packets go the right way. But everything else is not getting picked up by the firewall/gateway group

So what on earth have I missed ? Do I still need to set a default gateway ?

B. Rgds
John

[AdSchellevis]

Hi John,

Yes, you need a gateway on both interfaces, then when your setup is functional you can begin setting up multiwan.
It's also good to check if the gateways report the correct status in System -> Gateways -> Status.

Policy based routing uses the gateway status to determine valid targets.

Best regards,

Ad

[reep]

Hi Ad,

me back playing again! I think I might have solved my original WAN issues with ISPs so testing other bits and pieces to see what I can make work

Yes, you need a gateway on both interfaces, then when your setup is functional you can begin setting up multiwan.

OK, I reset the interface IPs manually and the gateways as well. I think it is safer than auto generated gateways.

I now get the fact that a default gateway must be set, but the weighting in the Gateway affects which route packets will take.

I think there is a bug here. If you set the WAN IPs manually the system forces you to have a default gateway, but if you use DHCP for both it does not! I can add a bug if required.

It's also good to check if the gateways report the correct status in System -> Gateways -> Status.

Yup - that seems OK but as the box is in the UK, and I am not, I need my young monkey to pull cables and make sure it falls over correctly :-) However as each WAN has the correct gateway I think it should be OK now. Will test tomorrow

Policy based routing uses the gateway status to determine valid targets.

OK. I think I get that now. For the benefit of others you need to set the Tier in Gateway Groups and the Weight in Gateway/Advanced (I think I am right in saying)


For reference here is the updated routing table which is the same bar the fact that WAN1/vtnet1 is set as default by the system.

Internet:
Destination        Gateway            Flags      Netif Expire
default            192.168.1.1        UGS      vtnet1
8.8.4.4            192.168.2.1        UGHS     vtnet3
8.8.8.8            192.168.1.1        UGHS     vtnet1
10.0.0.0/24        link#1             U        vtnet0
10.0.0.251         link#1             UHS         lo0
127.0.0.1          link#8             UH          lo0
192.168.1.0/24     link#2             U        vtnet1
192.168.1.11       link#2             UHS         lo0
192.168.2.0/24     link#4             U        vtnet3
192.168.2.11       link#4             UHS         lo0


To test each route works you can traceroute the Gateway monitoring IP and see that the packet goes out the correct route

root@OPNsense:~ # traceroute -n 8.8.8.8
traceroute to 8.8.8.8 ( 8.8.8.8 ), 64 hops max, 40 byte packets
 1  192.168.1.1  0.389 ms  0.562 ms  0.291 ms
 2  81.139.192.1  27.124 ms  27.078 ms  27.322 ms
.....

root@OPNsense:~ # traceroute -n 8.8.4.4
traceroute to 8.8.4.4 ( 8.8.4.4 ), 64 hops max, 40 byte packets
 1  192.168.2.1  0.433 ms  0.525 ms  0.298 ms
 2  81.139.96.1  27.452 ms  25.104 ms  25.335 ms
.....

Beyond that I am not sure how else to tell if the balancing is working correctly and the system is using both routes ?

B. Rgds
John

[reep]

OOohhh nasty gotcha.

If I set up a IPSEC connection and specify WAN3 the system still creates a route via WAN1

Here's the routing table for it:

192.168.97.0/24    192.168.1.1        US       vtnet1

I think it should be:

192.168.97.0/24    192.168.2.1        US       vtnet3

Here's the same connection now set to use the gateway group and it auto selects WAN1:

192.168.97.0/24    192.168.1.1        US       vtnet1

With it set on WAN1 I can ping from the remote box -> opnsense after creating a Pass rule, but not vice versa ! See my previous on the subject: https://forum.opnsense.org/index.php?topic=4217.0

(bangshead)

 :-)

[reep]

OOohhh nasty gotcha.

If I set up a IPSEC connection and specify WAN3 the system still creates a route via WAN1

https://github.com/opnsense/core/issues/1337