MultiWAN Load Balancer with mixed types of gateways (RFC1918+PPPoE)

[svenny]

Hi all,

I am experiencing connection problems trying to get load balancing between 2 gateways: 1 PPPoE and 1 RCF1918.

These are my gateways:

Name                 Interface    Protocol    Priority                     Gateway    Monitor IP    
PPPOEGW (active)    PPPINT    IPv4    253 (upstream)    1.2.3.4            1.1.1.1    
RFC1918GW            RFCINT    IPv4    255 (upstream)    192.168.8.1    8.8.8.8

This is my group of gateways:

Group Name    WANGWGROUP

Gateway            Tier
PPPOEGW          1
RFC1918GW      1

Trigger Level  Member down

Sticky connections are enabled under "Firewall->Settings->Advanced" and I've got the rule for DNS on LAN tab as the first rule.

In "System->Settings->General" I have got this:

DNS Server
1.1.1.1 PPPOEGW
8.8.8.8 RFC1918GW

and this:

Gateway switching    Allow default gateway switching  (enabled)

Then I have set the Gateway field to WANGWGROUP for the rule "Default allow LAN to any rule".

Now when I navigate the web I'm experiencing strange issues: sometimes it works, sometimes it timeouts...

Is there anything I'm missing with this setup? How could I troubleshoot this problem?

Versions:

OPNsense 21.7.5-amd64
FreeBSD 12.1-RELEASE-p21-HBSD
OpenSSL 1.1.1l 24 Aug 2021

Thank you in advance.

Cheers,
Svenny

[svenny]

Forgot to say that the gateways work perfectly using policy based routing.

Cheers,
Svenny

[va176thunderbolt]

Check your sticky settings under Firewall>Advanced>Multi-wan. You may want to experiment to see what provides you the most stability.

[FullyBorked]

I think @va176thunderbolt has the right idea.  Sticky connections can cause some issues they did me, for me it was mainly failing back to my primary when it was available again. 

Another thing to note, WAN failover isn't always seamless.  Some sites/services hate to have traffic spontaneously coming from another location maybe even taking a different route.  If you fail over and give it a few minutes and if everything works as expected, I might lean toward it's just the behavior of the failover.

[svenny]

Thanks for your advices. I disabled "Sticky connections" and now I have load balancing working. Some kind of speedtest is doubling my bandwidth speed while others show only one link, but I think it depends on the speedtest page.

I had some problems with VoIP, so I forced it on a specific gateway and now it works well.

I'm wordering if I could have problems with OpenVPN disabling "Sticky connections" in global settings... It seems to be working nicely till now.

Cheers,
Svenny