Messages

Ok. According to the manual "Do not use the system nameservers option if you have a multi-WAN setup and have Unbound running alongside multiple DNS servers configured in General with separate gateways assigned to them. Unbound will use the locally created routes to reach the system nameservers, which will not work when the gateway is down."
I guess adding nameservers to System: Settings: General is not the way or does it mean if I have selected separate gateways for each nameserver?

Anyway, I have added 1.1.1.1 and 1.0.0.1 as custom forwarding in "query forwarding". Tried both with and without nameservers put in (system-settings-general). I also added a firewall rule on WAN: IPv4 TCP/UDP, source this firewall, destination any on DNS port 53, as some have suggested. Same problem still occurs, domains work for about 10 secs than dns stops working.

I enabled query forwarding and DNS over TLS using 8.8.8.8, 8.8.4.4. Also added custom forwarding:
8.8.8.8 port 853 
8.8.4.4 port 853
Unfortunately this did not help, same problem occurs

I have two opnsense servers (VM on proxmox). Both are connected to fiber isp. They are connected in CARP IP for continous internet connection when one of the servers is down for maintenance. This works perfectly. I want multi WAN setup so mobile can take over if my fiber isp is down.
I use  Dnsmasq DNS & DHCP only for DHCP and haven`t set up DNS.
Unbound forwarding mode: do you mean query forwarding? It is disabled (disabled "use system nameservers"). Network interfaces: LAN, outgoing network interfaces:all. Disabled Enable DNSSEC Support.

System: Settings: General
DNS server: all boxes are blank.Use gateway: none

Disabled "Allow DNS server list to be overridden by DHCP/PPP on WAN"
Disabled "Do not use the local DNS service as a nameserver for this system"
Enabled "Allow default gateway switching"

WAN1 is my main fiber and WAN2 is netgeaer MR5200 mobile router (in passover mode)
I have also setup Unbound DNS, query forwarding is disabled, and dns server in system-settings-general are empty. Gateway switching is checked.

DNS works over WAN1, but when I disconnect WAN1 and WAN2 takes over, i can access external webpages for about 10 sec, then all new pages times out. I am able to ping external ip adresses as well as ip tv is still streaming seamlessly.

I have now spent several hours for many weeks trying to fix this but nothing seems to help. I suspect that the problem lies with Unbound DNS. When I manually change dns on my laptop to 8.8.8.8, DNS finally works but I don`t want to change to 8.8.8.8 on all may clients. I want to use my firewall DNS 192.168.50.1.

I also use Dnsmasq DNS & DHCP where DNS and gateway are directed to CARP IP on my firewall 192.168.50.1.

I setup failover by using the guide on https://docs.opnsense.org/manual/how-tos/multiwan.html as well as troubleshooting using chatgpt. I have read multiple posts here where people seem to have simlar problem.
OPNsense 25.7.11_2-amd64.

Anyone know what might be the problem?

Thanks for all your help. It has been a long learning journey for me:)

I just wanted to give an update. I ended up using gateway groups instead of frr. WAN on node1 as tier1, node2 (LAN) as tier2. In addition I added a 5g mobile usb router as tier3. If WAN fails, I still have internet routing through node 2, and if both WAN on node 1 and 2 is down, I still have net through the mobile router. If I shut down node 1, node 2 becomes master. So at the moment it seems to work as I hoped. I guess my system is more or less failsafe now without frr.

Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            81.xxx.xxx.1       UGS             re0
8.8.8.8            81.xxx.xxx.1       UGHS            re0
10.10.10.0/24      link#7             U               wg0
10.10.10.1         link#3             UHS             lo0
10.10.10.2         link#7             UHS             wg0
10.10.10.3         link#7             UHS             wg0
10.10.10.4         link#7             UHS             wg0
81.xxx.xxx.0/21    link#1             U               re0
81.xxx.xxx.1       link#1             UHS             re0
81.xxx.xxx.185     link#3             UHS             lo0
127.0.0.1          link#3             UH              lo0
192.168.50.0/24    link#2             U            vtnet0
192.168.50.1       link#3             UHS             lo0
192.168.50.2       link#3             UHS             lo0

Unplugged:
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
10.10.10.0/24      link#7             U               wg0
10.10.10.1         link#3             UHS             lo0
10.10.10.2         link#7             UHS             wg0
10.10.10.3         link#7             UHS             wg0
10.10.10.4         link#7             UHS             wg0
127.0.0.1          link#3             UH              lo0
192.168.50.0/24    link#2             U            vtnet0
192.168.50.1       link#3             UHS             lo0
192.168.50.2       link#3             UHS             lo0

I ran all configs and logs into AI. I dont know how accurate this is but:
"Your OSPF daemon is learning routes, but not injecting them into the FreeBSD kernel routing table.
That's why you don't see a second default route from the peer node."

Some more

Some of my configs

Running NIC in passthrough on both nodes. Interface go down as it should when I disconnect the cable. When running netsstat when disconnected, the WAN device dissappears from the routing table. It is present when wan is connected. I`ve read somewehre that some people experience a problem with default route persisting when upstream gateway is down.

Ahh, I see. I thought i mentioned that I run it in proxmox somewhere earlier. Thanks for clearing it up. I may try to run pci passthrough for WAN.

Oh sorry. :)
It does not seem to lead to the backup node 192.168.50.3 when disconnected.

Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            151.130.80.1       UGS          vtnet0
8.8.8.8            151.130.80.1       UGHS         vtnet0
10.10.10.0/24      link#7             U               wg0
10.10.10.1         link#3             UHS             lo0
10.10.10.2         link#7             UHS             wg0
10.10.10.3         link#7             UHS             wg0
10.10.10.4         link#7             UHS             wg0
127.0.0.1          link#3             UH              lo0
192.168.50.0/24    link#2             U            vtnet1
192.168.50.1       link#3             UHS             lo0
192.168.50.2       link#3             UHS             lo0
151.130.80.0/20    link#1             U            vtnet0
151.130.84.90      link#3             UHS             lo0

Internet6:
Destination                       Gateway                       Flags         Netif Expire
::1                               link#3                        UHS             lo0
fe80::%lo0/64                     link#3                        U               lo0
fe80::1%lo0                       link#3                        UHS             lo0

I think so. Here are some more screenshots that might help. (Black theme is node1 192.168.50.2, white is node 2 192.168.50.3.)
Routing table:

Thanks!
I  have tried different configurations but for some reasopn I can`t get it to work. The nodes do communicate with each other, master state is "Full/DR" and backup is "Full/Backup", however OSPF does not respond if WAN gateway  is down. When unplugging, gateway is down within just a few seconds. These are my configurations (neighbors, prefix lists and route maps are empty). I also turned off BFD until I get OSPF working. I have no gateway groups. My CARP VIP LAN ip is 192.168.50.1. Master router: 192.168.50.2, backup is 192.168.50.3. I have tried both with carp failover/demote, but that neither worked. So my current configuration avoids CARP so node1 always is master.


My ISP gateway ip