Messages

I think I figured out what the issue is/was.

Routes are setup for the gateway monitoring IPs. These same IPs are set in DNS servers in the General Configuration.

WAN1 which has a gateway monitor of 8.8.8.8 and WAN 1 DNS is also 8.8.8.8
WAN2 which has a gateway monitor of 1.1.1.1 and WAN 1 DNS is also 1.1.1.1

If a client(s) is on WAN2 they can't use google as their DNS server (as it's going out WAN1) I can't even ping 8.8.8.8.

I've used 4.2.2.1, 4.2.2.2, and 4.2.2.6 as the monitoring IPs (which routes now show in the routing table)...

So, making sure the gateway monitor is not the same as the DNS might have resolved this...

Is it just me or does WAN Balancing not work?

Failover seems to work fine, but if I put two WAN connections in Tier 1 then I get constant DNS or page load failures - having to refresh the page to get it to load. If I switch to failover it works fine.  (WAN1 on Tier 1 and WAN2 on Tier 2 (also tried reversing this to verify it's not a WAN connection issue).

I've followed the guide to a tee and also tried a fresh install on new hardware (previously Hyper-V VM and now on a qotom PC). 

1. Gateway groups created (packet loss and latency failover)
2. DNS pointing to each separately
3. Monitoring to google and cloudflare dns
4. Firewall points to gateway group
5. DNS firewall rule created for firewall
6. Using Unbound DNS (I've tried to focus on one WAN as well as all interfaces)
7. Default gateway switching is enabled.
8. Sticky connections enabled
9. Running the latest update (20.1.7)

I've also had to create traffic shapers to avoid bufferbloat; aside from that (and 1 VLAN) it's a fresh/default installation.

My connections are:
WAN1. LTE ISP CE (they have a router and give me a 192.168.209.x address on wan interface)
WAN2. LTE ISP CE (same ISP but give a 192.168.0.x address on wan2 interface)
(WAN 3 is not in balancing for groups above (it's in failover only))
WAN3. PPPoE ISP (modem has router that does the PPPoE connections and I've assigned 192.168.100.x address to the wan3 interface)

Questions:
1. Should I be selecting Upstream Gateway in the Single Gateway for any of these WAN single Gateways?
2. Far Gateway?
3. Priority (They were all the same - I changed values and seem to have better results - WAN3 was showing as default gateway - now it's on the bottom)).

I've had to resort to making an alias of half the IPs and put them in groups - so one group is in WAN 1 and everyone else is on WAN 2. I want to load-balance everyone though - this doesn't seem like a resolution....

I have 3 WAN connections - and trying to stick a few clients to only use one WAN. I was going to trace etc to verify this but curious:

I'm using Unbound and DHCP on OPNsense, all clients point to OPNsense for DNS. I have a firewall rule in first position for DNS with the destination being the firewall (OPNsense) and default gateway.

In the 2nd rule I have a Source of 1 client and the gateway forced to a Gateway group of only 1 WAN.

When this client does goes to a website it works and they get the WAN IP of that gateway.

OPNsense does the dns querisr for this client - what WAN does it use for these DNS queries? Does it use the same gateway the client uses or just whatever gateway?

I had a weird issue where I was having latency on one connection (not enough for failover) - this is noticeable in gaming and opening websites for all clients. However, the client forced on another WAN was also experiencing latency.  Which makes me believe that OPNsense was using this WAN for DNS but forcing other traffic for the specific client to the other WAN...  Would that be the case or may I have something else going on? How to I force DNS/all queries to go out the same WAN connection that they are ruled for?

You haven't provided much info on your setup; to be able to assist much...

I'm experiencing similar issues. (see thread: https://forum.opnsense.org/index.php?topic=17116.0 ) - This may give some places to look at least.

I'm curious what your setup is  --  to compare; maybe a common issue or setting's we've both missed on a 'new' install.

I tried upping the [Firewall - Settings - Advanced] source tracking timeout to 3000 and same issue... easy to test by browsing facebook - videos start playing when you scroll and if you watch for a second or two then you get the loading screen... Does anyone else get this with load balancing?

It may be a DNS issue as well...

I have OPNSense as the DNS Server using Unbound DNS - all clients point to OPNSense for DNS.

I have a rule to allow DNS as per (https://docs.opnsense.org/manual/how-tos/multiwan.html):
  IPv4 TCP/UDP   *   *   10.10.10.1   53 (DNS)   *   *   Local Route DNS

In [Services - Unbound DNS - General] Outgoing Network Interfaces is currently set to [All(recommended)].

Would this cause issues if DNS lookups went out 1 WAN while the traffic, for a session, went out another?

With DUAL WAN / Gateway Groups

1. Is there anyway to get the [Reporting - Traffic] reports to show what gateway each LAN IP/HOST is using - live?
        In [Interfaces]
        - if I choose LAN - I can see all the IPs - however this is combined traffic on both Gateways
        - I can uncheck all interfaces but the two WAN's in the graph - can I do the reverse? (can I select one host
           and the graph update?)

2. Is there a dashboard / report that shows both gateway live traffic in number value (instead of the graph)?
        In [Interfaces]
         - if I choose WAN1 - I only see the traffic for this WAN (obviously) - I'd like a way to see both at once
           (and on dashbord/custom page if possible)

2a. [Reporting - Traffic | Dashboard widget ] Is there a way to save the graph with only the gateway's selected? (instead of clicking all other interfaces to turn them off?)

Just switched to OPNsense from Untangled. So far I'm preferring OPNsense but I'm having issues with the dual wan balancing; which I wasn't having with Untangled.

While enabled I have clients (home users) that get pauses after most things they do. For example - a Amazon Firestick will show 'no internet' but connected to wifi after every stream or when going back to the main menu. After a few seconds it starts again. It appears to be switching connection on every session.  Similar issues with Google Home, PCs browsing facebook etc... no internet then internet then no internet.

When Gateway groups are setup as Tier 1 / Tier 1 (issue above is noticed).
When Gateway groups are setup as Tier 1 / Tier 2 (issues above go away).
When Gateway groups are setup as Tier 2 / Tier 1 (issues above go away).
I can force gateway on an IP and I have no issues at all (not using gateway groups).
Gateway Group Trigger is [Member Down]
Gateway monitoring is enabled (no check in [disable gateway monitoring]) and all are showing online.

Firewall - Settings - Advanced - [Use sticky connections] is checked.

I'm not sure what I'm missing here...