Local routing in Multi-WAN Setup

[ad]

Hi guys, thanks for this fine project!

I have been working with OPNsense for some time now, have 4 installations in productive use and have the following problem with a larger installation:

There is a HA setup with carp IPs and a multi WAN connection. The WAN connection consists of 3 ADSL routers, which should handle load balancing and failover equally. The routers are located in a transfer network in which the OPNsense boxes run with their common carp IP. Something like that:

Code: [Select]

     ADSL1                    ADSL2                    ADSL3
       :                        :                        :
       :                        :                        :
   .---+---.                .---+---.                .---+---.
   |  NAT  |                |  NAT  |                |  NAT  |
   '---+---'                '---+---'                '---+---'
       |                        |                        |
192.168.162.251          192.168.162.252          192.168.162.253
       |                        |                        |
       |        .--------------------------------.       |
       +--------|          TRANSFER NET          |-------+
                '---+------------------------+---'
                    |                        |
                    |                        |
               .----+-----.             .----+-----.
               | OPNsense |    carp     | OPNsense |
               '----+-----'             '----+-----'
                    |                        |
                    |                        |
                .---+------------------------+---.
                |              LAN               |
                '--------------------------------'

The gateways have the same weighting and are combined in a group with identical tiers, routed connections work as they should with policy based routing.

Now I have noticed that the local outgoing connections always run through the first gateway, no matter which default route is set and regardless of whether e.g. a host route is set. So my GW monitoring pings all run over the first GW, which is a bit stupid.

As a simple test I have set hostroutes to different ipinfo.io servers (fortunately, they have 4):

Code: [Select]

ad@fw01:~ % host ipinfo.io
ipinfo.io has address 216.239.32.21
ipinfo.io has address 216.239.34.21
ipinfo.io has address 216.239.36.21
ipinfo.io has address 216.239.38.21

some routes:

Code: [Select]

ad@fw01:~ % netstat -rnW | grep 216.239
216.239.32.21/32   192.168.162.251    UGS           0   1500 igb0_vlan11
216.239.34.21/32   192.168.162.252    UGS           0   1500 igb0_vlan11
216.239.36.21/32   192.168.162.253    UGS           0   1500 igb0_vlan11

curling from 3 ipinfo IPs:

Code: [Select]

ad@fw01:~ % curl --resolve ipinfo.io:80:216.239.32.21 ipinfo.io/ip
80.xxx.xxx.xxx
ad@fw01:~ % curl --resolve ipinfo.io:80:216.239.34.21 ipinfo.io/ip
80.xxx.xxx.xxx
ad@fw01:~ % curl --resolve ipinfo.io:80:216.239.36.21 ipinfo.io/ip
80.xxx.xxx.xxx

In my opinion I should receive my 3 external ADSL addresses, but I always get the ones from the first gateway. I have not configured local outgoing policy based routing, that would be nonsense, because that would overwrite the host routes for the gateway monitoring.

I would be interested to know how it runs with local static default routes now (keyword local services). Why can't I set multiple default routes with the same metric? Would there not round-robin be used? If a GW fails, the route is taken from the routing table, done.

Currently, under "System > Gateways > Single" only one default gateway can be set, but if I remove that (default), it will remain in the routing table and will not be replaced until the next reboot. Is there a reason, or is that a bug?

Okay, I'd be happy if you'd give me a hint on how to get a handle on the gateways for local connections. It's probably just a kernel setting.

Thanks for everything!

Greetings
AD

[franco]

Hi there,

So first thing to note is that policy routes will completely bypass system routes every time.

Secondly, you may want to unset the sticky load balancing, but it will probably cause issues that way (it's only by default since at least 18.1 I think unless your first install was before that). So yes it distributes round-robin, but any give source IP will stick to a particular WAN for a period of time unless forced away due to connectivity issues.

The default gateway will stick around, that's true. Mainly because there is no "flush default route" feature, only a "set default route to xxx" undoing the previous one in the process.

If you want flexible / special routes you need to add them to your firewall rules as specialised policy routes.

Hope this helps.


Cheers,
Franco