New WAN failover configuration sort of works

[bob9999]

I have a problem setting up wan failover / load sharing.

Specifically, it seems to work acceptably when I leave a few steps out but dns fails if I follow all the steps that are commonly recommended.

I can't even log into 192.168.1.1:3000 (adguard home) when the final steps are attempted. Although 192.168.1.1 for Opnsense access works normally.

I have 2 gateway groups, Failover and LoadBalance. I have 2 but will probably settle on LoadBalance. Monitor IP addresses are correct.

Manually shutting one down fails over correctly to the other. Setting both WAN priorities the same seems to let the router decide. LoadBalance is RoundRobin Sticky on the gateway group. Currently, WAN1 is priority 254 and WAN2 is priority 255. 'Allow default gateway switching' makes no difference checked or unchecked.

As soon as I put a gateway group, either one, in the LAN rule everything chokes. No dns. Nothing gets fixed if I add a DNS rule above the LAN gateway rule.

Everything springs back to life when I remove the Gateway from the LAN.

Along the way I discovered DNS speeds up considerably when I stopped using unbound and allowed Adguard Home to be the sole DNS forwarder. That configuration stays. Unbound and DNSMasq are both disabled. I have some DNS servers also listed on the main system general page, FWIW.

Internets are Comcast and T-Mobile 5G. CGNAT from TM appears to work great double natted. T-Mobile is 192.168.12.1 from them as WAN2 address.

Question: Should I leave well enough alone? Or is there a way to make all the known and recommended configuration steps work as described in almost every article I can find on the subject?

[viragomann]

As soon as I put a gateway group, either one, in the LAN rule everything chokes. No dns. Nothing gets fixed if I add a DNS rule above the LAN gateway rule.

Seems you made the same mistake as another guy recently. Look here: Setting up WAN failover

[bob9999]

Thanks. I read this as try it again and then reboot. Ok.

Edit.

I played around with the dns rule and it finally accepted the /32 address without the gui telling me it didn't exist. I have the rule set to inactive until I have the time to go at it again.  Maybe Thursday.

[bob9999]

I fixed the adguard home access problem by editing its configuration file, adguardhome.yaml. I gave it a different administrative port. The accepted standard is port 3000. Apparently, that port conflicts with some aspect of the setup of gateway groups, according to some search engine research. Wikipedia says the port number I gave it is unused.

The DNS LAN rule mentioned in the reference post above was already above the LAN rule in question. It was originally very difficult to create because Opnsense kept telling me that 192.168.1.1/32 did not exist. On a random follow up attempt it accepted the address. I was trying to follow the example in the Opnsense documentation.

It still fails with DNS access. Even after a router reboot. Everything returns to life after I back out the changes to the rules.

Although Adguard Home as a DNS forwarder works very well. My DNS queries are in the 12 ms range consistently.

Using Unbound as a forwarder to the DNS servers on the main system setup page, the queries were in the 20 ms range or slightly better.

Using Unbound only, the queries were in the 30 ms to 50 ms range. That was a surprise as I always believed Unbound was a sort of superior standard.

On the plus side, ignoring Unbound on an Adguard Home install, leaving Unbound disabled, should make Adguard Home a no brainer to install. Getting Adguard Home to coordinate with Unbound was also a frustrating problem. I fail to see how Unbound is 'superior', which I also believed until I went through all this.

I will keep playing with the Failover / LoadBalance configurations and eventually figure it out.

DNS access is the problem. Advice from people who have already went through this is wanted.

Quick Edit:

Could someone tell me where the graph is that shows historical traffic levels through the WAN interfaces? I saw one that showed how much data was going through WAN1 and WAN2, but I can't seem to find it again. Is this my imagination?

[nero355]

I fixed the adguard home access problem by editing its configuration file, adguardhome.yaml. I gave it a different administrative port. The accepted standard is port 3000. Apparently, that port conflicts with some aspect of the setup of gateway groups, according to some search engine research. Wikipedia says the port number I gave it is unused.

Technically port 3000 is in the so called 'Registered Range' and only the ports above 49152 are all completely 'Dynamic' so that sort of thing can happen indeed...

It still fails with DNS access. Even after a router reboot. Everything returns to life after I back out the changes to the rules.

Although Adguard Home as a DNS forwarder works very well. My DNS queries are in the 12 ms range consistently.

Using Unbound as a forwarder to the DNS servers on the main system setup page, the queries were in the 20 ms range or slightly better.

Using Unbound only, the queries were in the 30 ms to 50 ms range.

That was a surprise as I always believed Unbound was a sort of superior standard.

Makes sense if you read this : https://docs.pi-hole.net/guides/dns/unbound/#what-is-a-recursive-dns-server
Unbound needs to contact all the DNS Servers involved with a specific domain and that takes some time the first time you query the domain :)

I fail to see how Unbound is 'superior', which I also believed until I went through all this.

IMHO you simply need to take it away from OPNsense and run everything on a seperate device : Raspberry Pi/Intel Atom NUC or Proxmox VM/LXC and you will be a lot happier :)

[bob9999]

nero355:

I want my DNS ad blocker on the router because that creates fewer potential failure points. I used run pihole long ago for a different router. I had 2 pihole servers on 2 PCs running in Hyper-V on Ubuntu Server. Windows knocked both off when they stopped a restart to advertise Windows 11. Lesson learned.

The ad blocker stay on the router. Otherwise there's no point in using OPNsense. My old ER605 V2 has some easy load balancing / failover double WAN setups built right in and VLANS are also much easier with TP-Link. Going that way adds failure points because of external ad blockers for DNS. Potential failure points need to be avoided. GLINet has 2 models, the Flint 2 and Flint 3, which both have enough RAM and storage to support a realistic Adguard Home server. The other models they sell are too small to hold any more than a light basic configuration. Mikrotik can support Adguard Home in a container, but I avoid those, they have fans but I only see unreliability if I look hard enough.

My thinking is if OPNsense offers a configuration like this one, they should be able to illustrate workable configurations. My setup is not at all a strange one. To my thinking, there should be a documentation page that deals with common problems like this. The DNS LAN rule illustration was fragmentary at best.

Wikipedia has a list of ports in use and possible use. If its not mentioned then it's OK to use. I chose 3100.


The big news is that it seems to work now for LoadBalancing. The LAN rule points to the LoadBalancing gateway group. A DNS rule was put above the LAN rule. There seemed to be an error in how the DNS rule was configured. There was not enough detail in the official OPNsense documentation.

Adguard home works on the network, but I can't get to the Adguard Home interface again, even with the new port. But I tested it and it works as an ad blocker on my TVs. So now, access to the Adguard Home interface is the new problem to fix. Help would be appreciated.

Quick Edit:

Everything appears to be working, finally.

The interface to Adguard Home appears to be working now. I'll edit this comment if anything material changes.

I copied the LAN DNS pass rule and modified it for port 3100 as opposed to 53. it works.

The OPNsense documentation works well, except for the DNS LAN rule. It needs to be reworked to reflect modern interfaces. A comment in the same section for alternate ports, such as 3000 for Adguard Home, should be added.

I have no idea how things would work if Unbound and an alternate port such a 5353 or 53530, for Adguard Home were a part of the configuration.