WAN Balancing Not working

[cinntech]

Is it just me or does WAN Balancing not work?

Failover seems to work fine, but if I put two WAN connections in Tier 1 then I get constant DNS or page load failures - having to refresh the page to get it to load. If I switch to failover it works fine.  (WAN1 on Tier 1 and WAN2 on Tier 2 (also tried reversing this to verify it's not a WAN connection issue).

I've followed the guide to a tee and also tried a fresh install on new hardware (previously Hyper-V VM and now on a qotom PC). 

1. Gateway groups created (packet loss and latency failover)
2. DNS pointing to each separately
3. Monitoring to google and cloudflare dns
4. Firewall points to gateway group
5. DNS firewall rule created for firewall
6. Using Unbound DNS (I've tried to focus on one WAN as well as all interfaces)
7. Default gateway switching is enabled.
8. Sticky connections enabled
9. Running the latest update (20.1.7)

I've also had to create traffic shapers to avoid bufferbloat; aside from that (and 1 VLAN) it's a fresh/default installation.

My connections are:
WAN1. LTE ISP CE (they have a router and give me a 192.168.209.x address on wan interface)
WAN2. LTE ISP CE (same ISP but give a 192.168.0.x address on wan2 interface)
(WAN 3 is not in balancing for groups above (it's in failover only))
WAN3. PPPoE ISP (modem has router that does the PPPoE connections and I've assigned 192.168.100.x address to the wan3 interface)

Questions:
1. Should I be selecting Upstream Gateway in the Single Gateway for any of these WAN single Gateways?
2. Far Gateway?
3. Priority (They were all the same - I changed values and seem to have better results - WAN3 was showing as default gateway - now it's on the bottom)).

I've had to resort to making an alias of half the IPs and put them in groups - so one group is in WAN 1 and everyone else is on WAN 2. I want to load-balance everyone though - this doesn't seem like a resolution....

[h.kluncker]

I have the same problem with a pretty similar setup.
Execpt with 2 DSL Lines as WAN, no shaping (yet), running 20.1.5

I followed the steps in the Multi WAN documentation, the steps (1-8) you mentioned above.

Same result as yours, each connection works as a single Gateway.
As soon as i direct the traffic through the Gateway Group with the Firewall Rule, half the the connection don't work.

I'm new to opnsense too and don't know how to troubleshoot this, so my Post is more a bump up.

[Singman]

I'm not sure how OPNSense is working but :
Using loadbalancing with 2 or more WAN need a stateful engine otherwise that will not work.
When a web connexion is made (web is a good example because it's doing many requests with a non-linear timeline), you can have packets going from either WAN but the firewall should remember the source and what WAN interface it's using. Otherwise, any destination with a security will detect a "spoofing" or 'man-in-the-middle" attack and will block you (or cookie based on public @IP will fail, etc...).
The hard part with web is to remember that "path" for a very long time because a user could stay on a page for an undefined period.

[h.kluncker]

Thanks for the reply.

According to the documentation opnsense is a stateful firewall.
As i understand it, "Sticky connections" is there to force established connections to use the same WAN.

The Problem occurs before any sort of idle-timeout.
On the second WAN no connection is established / routes to nowhere (as far i can tell). Any idea how to verify/troubleshoot that assumption?

[cinntech]

I think I figured out what the issue is/was.

Routes are setup for the gateway monitoring IPs. These same IPs are set in DNS servers in the General Configuration.

WAN1 which has a gateway monitor of 8.8.8.8 and WAN 1 DNS is also 8.8.8.8
WAN2 which has a gateway monitor of 1.1.1.1 and WAN 1 DNS is also 1.1.1.1

If a client(s) is on WAN2 they can't use google as their DNS server (as it's going out WAN1) I can't even ping 8.8.8.8.

I've used 4.2.2.1, 4.2.2.2, and 4.2.2.6 as the monitoring IPs (which routes now show in the routing table)...

So, making sure the gateway monitor is not the same as the DNS might have resolved this...

[h.kluncker]

Interesting Point.

If I use the same DNS-Servers as monitoring IP they must match, since they both set a Route to the specified IP with the corresponding gateway.
I might have messed something up there, will check that as soon i get on site.

Which also means, if one of the Clients has an fixed DNS entry of one of those DNS/monitoring-IPs, there will be trouble too.

[mscd]

Hello folks,

I am observing a similar behavior/problems with OPNsense 21.1.7, although I already deactivated uplink monitoring of WAN-interfaces completely (e.g. per ping to 1.1.1.1 or 8.8.8.8 ).
WAN-LoadBalancing is not working properly ... half of the connections get stuck ... by using one WAN-interface alone, everything seems to be fine.

Any further advice/hints to this problem?

Best regards,
mscd

[mscd]

Update: With "sticky connections" turned off, client connections seem to work stable ... on the other hand, I thought one should activate "sticky connections" in the WAN-balancing use-case due to the possibility of https-session-problems.

So what?

[thomas7467]

Hi,

Did you finally get the loadbalancing working?

I have a similar configuration :

1. Gateway group created (member lost) with 2 WAN interfaces;

2. Different Google DNS configured for each WAN Interface 8.8.8.8 for the first, 8.8.4.4 for the second;

3. Monitoring interfaces enabled (4.2.2.1 and 4.2.2.2 IP)

4. Defaut LAN Firewall rules points to gateway group previously created;

5. DNS firewall rule created for firewall (disabled cause I use External DNS 8.8.8.8 and 8.8.4.4 in my DHCPv4 configuration);

7. Allow Default gateway switching is enabled;
8. Sticky connections enabled

9. Running the 21.7.3_3 release.

Works fine in failover mode, but loadbalancing mode get web traffic into issues (DNS errors, slow browsing...)

Any ideas of what could be wrong?

[KatiaSisHost]

Same here in my case, work some hours and after.. packet loss and disconnection intermitents

[ipzipzap]

Hi,

since about two weeks I am trying to get load balancing to work, but I am experiencing exactly the same problems as described here. Failover works like a charm, load balancing is a big mess. I tested many options and tips and I reinstalled and rebuild from scratch countless times.

Any help would be greatly appreciated.

Thanks!

[svenny]

Hi, I had the same issue and disabling "Firewall->Settings->Advanced->Multi-WAN->Sticky connections" solved the problem. No issues with HTTPS sites.

Cheers,
Svenny

[rberger]

Looks like if you go to Firewall->Settings->Advanced  and enable  Sticky Connection  AND DISABLE  Shared forwarding Multi-wan will work with Sticky Connection.

Looks like you can't have Sticky Connection AND (Shared Forwarding OR Captive Portal)

As per:
https://forum.opnsense.org/index.php?topic=17116.msg93965#msg93965
https://forum.opnsense.org/index.php?topic=19977.msg92543#msg92543

[buecker]

Based on how many views here and how many threads on Reddit there seems to be a consensus that load balancing is challenging.  There must be an answer to this.  I have been battling these issues for many months and haven't found any definite answer. 

Failover works great it is just the load balancing that refuses to work. I am in desperate need of load balancing but when I do get this figured out I am posting as many screenshots as I can!

[mostlyharmless]

I wanted to add my two cents into this troubleshooting effort - I was having similar issues in getting load balancing to work - nothing seemed to do the trick but like everyone here, failover seemed fine.

However, I did get load balancing working - there was one checkbox that is absent from the instructions that I think was the key - when you create the new DNS firewall rule I also checked "Quick" (apply the setting immediately on match). After that, DNS resolution worked just fine with load balancing setup as per opnsense's documentation.

My theory is that if you don't select that box, the rest of the firewall rules get applied and something overwrite the DNS outbound rule or messes up routing somehow.

Hope this helps - please post a reply if this also fixed your load balancing issue.