Topics

1

18.1 Legacy Series / Local routing in Multi-WAN Setup

« on: April 16, 2018, 10:16:53 pm »

Hi guys, thanks for this fine project!

I have been working with OPNsense for some time now, have 4 installations in productive use and have the following problem with a larger installation:

There is a HA setup with carp IPs and a multi WAN connection. The WAN connection consists of 3 ADSL routers, which should handle load balancing and failover equally. The routers are located in a transfer network in which the OPNsense boxes run with their common carp IP. Something like that:

Code: [Select]

     ADSL1                    ADSL2                    ADSL3
       :                        :                        :
       :                        :                        :
   .---+---.                .---+---.                .---+---.
   |  NAT  |                |  NAT  |                |  NAT  |
   '---+---'                '---+---'                '---+---'
       |                        |                        |
192.168.162.251          192.168.162.252          192.168.162.253
       |                        |                        |
       |        .--------------------------------.       |
       +--------|          TRANSFER NET          |-------+
                '---+------------------------+---'
                    |                        |
                    |                        |
               .----+-----.             .----+-----.
               | OPNsense |    carp     | OPNsense |
               '----+-----'             '----+-----'
                    |                        |
                    |                        |
                .---+------------------------+---.
                |              LAN               |
                '--------------------------------'

The gateways have the same weighting and are combined in a group with identical tiers, routed connections work as they should with policy based routing.

Now I have noticed that the local outgoing connections always run through the first gateway, no matter which default route is set and regardless of whether e.g. a host route is set. So my GW monitoring pings all run over the first GW, which is a bit stupid.

As a simple test I have set hostroutes to different ipinfo.io servers (fortunately, they have 4):

Code: [Select]

ad@fw01:~ % host ipinfo.io
ipinfo.io has address 216.239.32.21
ipinfo.io has address 216.239.34.21
ipinfo.io has address 216.239.36.21
ipinfo.io has address 216.239.38.21

some routes:

Code: [Select]

ad@fw01:~ % netstat -rnW | grep 216.239
216.239.32.21/32   192.168.162.251    UGS           0   1500 igb0_vlan11
216.239.34.21/32   192.168.162.252    UGS           0   1500 igb0_vlan11
216.239.36.21/32   192.168.162.253    UGS           0   1500 igb0_vlan11

curling from 3 ipinfo IPs:

Code: [Select]

ad@fw01:~ % curl --resolve ipinfo.io:80:216.239.32.21 ipinfo.io/ip
80.xxx.xxx.xxx
ad@fw01:~ % curl --resolve ipinfo.io:80:216.239.34.21 ipinfo.io/ip
80.xxx.xxx.xxx
ad@fw01:~ % curl --resolve ipinfo.io:80:216.239.36.21 ipinfo.io/ip
80.xxx.xxx.xxx

In my opinion I should receive my 3 external ADSL addresses, but I always get the ones from the first gateway. I have not configured local outgoing policy based routing, that would be nonsense, because that would overwrite the host routes for the gateway monitoring.

I would be interested to know how it runs with local static default routes now (keyword local services). Why can't I set multiple default routes with the same metric? Would there not round-robin be used? If a GW fails, the route is taken from the routing table, done.

Currently, under "System > Gateways > Single" only one default gateway can be set, but if I remove that (default), it will remain in the routing table and will not be replaced until the next reboot. Is there a reason, or is that a bug?

Okay, I'd be happy if you'd give me a hint on how to get a handle on the gateways for local connections. It's probably just a kernel setting.

Thanks for everything!

Greetings
AD