DNS on Multi-WAN failover does not work - any tips? Now with logfile...

[lar.hed]

I seem to have a problem with my Multi-WAN failover....

Gateway works just fine, but DNS does not - if main WAN (FTTH) goes down, then DNS stops to work.

Yes I have the port 53 rules in place (on my Alias for ALL_LAN), however when I started all this Multi-WAN failover in the start, I did not use Alias - now I moved on, and close to only use Alias - and now DNS does not work when failing main WAN.

Now this could most likly be because I overlooked something (again  ) and yes I am still learning.

Or Alias does not play well with failover... Which I do not believe in... Yet... 

And yes I use Unbound Plus DNS-over-TLS - so I like to route all DNS queries over DoT.

This I can say: With WAN_FTTH online - everythign works just as planned. With only WAN_LTE (backup WAN if you like) gateway for sure seems to work, and from within OPNsense I can do DNS lookup (Interface -> Diag -> DNS Lookup) that works over DoT (Unbound Plus settings work - yea!). I can also, in the firewall log notice that there seem to be traffic going against the local DNS server thru the Firewall Rule. Yet no connection in my clients web browsers...

Any ideas are more than welcome :-)

[lar.hed]

Okay, so I am diging around in the system, and decided to look deeper into the system log, and the thing I don't get is why there is only configure steps when WAN-FTTH is online again - why is there no steps like this when WAN-FTTH goes offline so the WAN-LTE (backup, or failover interface) get's the same treatment?

Code: [Select]

2020-08-08T11:48:41 kernel: pflog0: promiscuous mode enabled
2020-08-08T11:48:41 kernel: pflog0: promiscuous mode disabled
2020-08-08T11:48:40 opnsense: /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway 'yy1.yy2.yy3.1'
2020-08-08T11:48:33 opnsense: plugins_configure dns (execute task : unbound_configure_do())
2020-08-08T11:48:33 opnsense: plugins_configure dns (execute task : dnsmasq_configure_do())
2020-08-08T11:48:33 opnsense: plugins_configure dns ()
2020-08-08T11:48:32 opnsense: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
2020-08-08T11:48:32 opnsense: plugins_configure dhcp ()
2020-08-08T11:48:32 opnsense: plugins_configure ipsec (execute task : ipsec_configure_do(,wan))
2020-08-08T11:48:32 opnsense: plugins_configure ipsec (,wan)
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.linkup: ROUTING: keeping current default gateway 'yy1.yy2.yy3.1'
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to yy1.yy2.yy3.1
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'
2020-08-08T11:48:32 dhclient[83580]: bound to yy1.yy2.yy3.26 -- renewal in 43200 seconds.
2020-08-08T11:48:32 opnsense: plugins_configure newwanip (execute task : webgui_configure_do(,wan))
2020-08-08T11:48:32 opnsense: plugins_configure newwanip (execute task : vxlan_configure_interface())
2020-08-08T11:48:32 opnsense: plugins_configure newwanip (execute task : unbound_configure_do(,wan))
2020-08-08T11:48:32 opnsense: plugins_configure newwanip (execute task : openssh_configure_do(,wan))
2020-08-08T11:48:32 opnsense: plugins_configure newwanip (execute task : opendns_configure_do())
2020-08-08T11:48:32 opnsense: plugins_configure newwanip (execute task : ntpd_configure_defer())
2020-08-08T11:48:32 opnsense: plugins_configure newwanip (execute task : dyndns_configure_do(,wan))
2020-08-08T11:48:32 opnsense: plugins_configure newwanip (,wan)
2020-08-08T11:48:32 opnsense: plugins_configure vpn (execute task : openvpn_configure_do(,wan))
2020-08-08T11:48:32 opnsense: plugins_configure vpn (execute task : ipsec_configure_do(,wan))
2020-08-08T11:48:32 kernel: pflog0: promiscuous mode enabled
2020-08-08T11:48:32 kernel: pflog0: promiscuous mode disabled
2020-08-08T11:48:32 opnsense: plugins_configure vpn (,wan)
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'yy1.yy2.yy3.1'
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: The WAN_DHCP6 monitor address is empty, skipping.
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 9.9.9.9 via xx1.xx2.1.1
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 9.9.9.9 via xx1.xx2.1.1
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 1.1.1.1 via yy1.yy2.yy3.1
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 1.1.1.1 via yy1.yy2.yy3.1
2020-08-08T11:48:32 opnsense: plugins_configure monitor (execute task : dpinger_configure_do())
2020-08-08T11:48:32 opnsense: plugins_configure monitor ()
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: creating /tmp/igb1_defaultgw using 'yy1.yy2.yy3.1'
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: removing /tmp/igb2_defaultgw
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to yy1.yy2.yy3.1
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
2020-08-08T11:48:32 opnsense: plugins_configure hosts (execute task : unbound_hosts_generate())
2020-08-08T11:48:32 opnsense: plugins_configure hosts (execute task : dnsmasq_hosts_generate())
2020-08-08T11:48:32 opnsense: plugins_configure hosts ()
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: On (IP address: yy1.yy2.yy3.26) (interface: WAN_FTTH[wan]) (real interface: igb1).
2020-08-08T11:48:32 opnsense: /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'igb1'
2020-08-08T11:48:32 dhclient: Creating resolv.conf
2020-08-08T11:48:32 dhclient: New Routers (igb1): yy1.yy2.yy3.1
2020-08-08T11:48:32 dhclient: New Broadcast Address (igb1): yy1.yy2.yy3.127
2020-08-08T11:48:32 dhclient: New Subnet Mask (igb1): 255.255.255.128
2020-08-08T11:48:32 dhclient: New IP Address (igb1): yy1.yy2.yy3.26
2020-08-08T11:48:32 dhclient: Comparing IPs: Old: yy1.yy2.yy3.26 New: yy1.yy2.yy3.26
2020-08-08T11:48:32 dhclient: Starting delete_old_states()
2020-08-08T11:48:32 dhclient[83580]: DHCPACK from yy1.yy2.yy3.1
2020-08-08T11:48:31 dhclient[83580]: DHCPREQUEST on igb1 to 255.255.255.255 port 67
2020-08-08T11:48:31 dhclient: Removing states from old IP 'yy1.yy2.yy3.26' (new IP '')
2020-08-08T11:48:31 dhclient: Comparing IPs: Old: yy1.yy2.yy3.26 New:
2020-08-08T11:48:31 dhclient: Starting delete_old_states()
2020-08-08T11:48:31 opnsense: /usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan
2020-08-08T11:48:31 opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan
2020-08-08T11:48:31 kernel: igb1: link state changed to UP


--- FTTH ONline


2020-08-08T11:46:12 kernel: pflog0: promiscuous mode enabled
2020-08-08T11:46:12 kernel: pflog0: promiscuous mode disabled
2020-08-08T11:46:12 opnsense: /usr/local/etc/rc.filter_configure: Ignore down inet6 gateways : WAN_DHCP
2020-08-08T11:46:12 opnsense: /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway 'xx1.xx2.1.1'
2020-08-08T11:46:12 opnsense: /usr/local/etc/rc.filter_configure: Ignore down inet gateways : WAN_DHCP
2020-08-08T11:45:58 kernel: pflog0: promiscuous mode enabled
2020-08-08T11:45:58 kernel: pflog0: promiscuous mode disabled
2020-08-08T11:45:58 opnsense: /usr/local/etc/rc.filter_configure: ROUTING: creating /tmp/igb2_defaultgw using 'xx1.xx2.1.1'
2020-08-08T11:45:58 opnsense: /usr/local/etc/rc.filter_configure: ROUTING: removing /tmp/igb1_defaultgw
2020-08-08T11:45:57 opnsense: /usr/local/etc/rc.linkup: Clearing states for stale wan route on igb1
2020-08-08T11:45:57 dhclient[91167]: exiting.
2020-08-08T11:45:57 dhclient[91167]: connection closed
2020-08-08T11:45:57 opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan
2020-08-08T11:45:57 kernel: igb1: link state changed to DOWN


--- FTTH OFFline



[lar.hed]

I'll continue my monolog then:

I decided to remove one of my interface groupings, the ALL_WAN. First I did take a backup, then I cloned all rules form ALL_WAN to first WAN_FTTH and then to WAN_LTE. Then I removed all rules for ALL_WAN, and edited my floating rules somewhat, and then removed my ALL_WAN interface group. That killed DNS completely... I could not even run diagnostics DNS Lookup - so I restored my backup. Back to kind-of-working-order except no failover DNS.

However I am a bit intrigued by the fact that running the rules on interfaces kills the DNS - however running the same rules on interface group works. What the heck happened here?

I am getting awfully close to deleting everything and start over - however I would soooo much learn first what I am doing wrong. The thing is, when I setup failover, I only had WAN_FTTH and LAN, and then added the WAN_LTE - everything worked. Now I have Alias and grouped interfaces (ALL_WAN and ALL_LAN) - and failover does not work for DNS (but Gateway works....).

[lar.hed]

Something is strange for sure - I decided to start clean, and even at only LAN and the two WAN interfaces attached, I could not get Multi-WAN Failover to work. This worked before when I started woth OPNsense so I know this should have worked.

I tried something else also: Erased ALL Firewall rules, port forward, alias, and interface groups (as close to an reset as possible except IDS was not removed, and maybe something else) - that killed DNS all together, nothing worked, not even diag DNS lookup.

So I am still at the same point: Failover gateway works, DNS does not work when failover.

So now I decided to upgrade to 20.7, still the same problem with failover - I can not find anything of what I might have done wrong. So I continue with the latest instead. I will continue to search for what I have done wrong....

[edz]

We’re you able to resolve this? I’m facing similar issues, DNS is not working when a gateway group is set on my firewall rules despite allowing DNS as the first rule per the WAN failover docs