Traffic from backup node getting "No route to host" after upgrading to 19.7.x

[Tsuroerusu]

I run a high availability setup with two OPNsense firewalls in failover mode. Each node only has a single WAN interface.

Yesterday, I decided to upgrade to 19.7.x (I was on 19.1.10), and like I usually do I start with the backup node. The first jump to 19.7 went fine, but when the upgrade was finished and the system had rebooted, and I tried to do the minor update to 19.7.3, I got "No address record found for the selected mirror.". I tried multiple different mirrors but with the same result. I eventually discovered that the system could not resolve DNS, and I then tried pinging 8.8.8.8 and I got "No route to host".

On Twitter @opnsense hinting at it being a multi-WAN issue that was fixed in 19.7.3, so eventually I ended up setting up a local OPNsense mirror, and having my backup node get the update from there, which installed fine. But after it had finished and rebooted, the problem persisted.

I have not yet upgraded the primary node, because if this problem were to also occur on that one, then I would bring myself into deep doo-doo.

What could be causing this? I saw something about gateway issues in other posts, and I tried enabling the option "Disable Force Gateway" under Firewall > Settings, but that did not resolve the issue.

[tong2x]

had issues with 19.7.3 initially, dunno if applicable to you.
i have 2 opnsense server 1 with dual wan and 1 with 1wan.
the wan1 upgraded without issues...

the multiwan, for some reason, got messed up in the routing I supposed.
choosing wan2 ad gateway will have no internet, the dashboard will report wan2 as connected/up
wan1 and default route is ok

in the end I, removed wan2 interface only to set it again... it now works.
point is... maybe there is a routing conflict in the config. since your setup is totally different not sure what could cause it.
it was a easy desicion for me to delete wan2 interface because it is not working, and not complicated that could affect my system. but it was sure worth while after setting it up again, it was working as it should.
routing to that gw works and forcing wan2 in firewall rules works.

maybe a reset of your interface and links might help

[Tsuroerusu]

!!! I FOUND THE PROBLEM !!!

It is 03:37, and I just wanted to let everybody know that I found the problem! 

So I just finished re-watch Interstellar, so I was feeling clever as it was, and I decided to look over the release notes for 19.7 and 19.7.3, and this time round the following items struck a cord they had not previously:

"Gateways influence default switching order by weight"

"o system: add defunct gateways to GUI in disabled state"

"o firewall: restore automatic outbound NAT pre-19.7 behaviour which excludes gateways not configured and not dynamic"

So I thought, "Wait a minute, I am getting 'No route to host' and here it says there were significant changes to something to do with gateways, I better investigate that part specifically".

And so I opened up the Web GUI on my backup node, and then I went into System --> Gateways --> Single.

I only have a single gateway and my ISP uses IPv4, and so it immediately struck me as odd that the gateway was saying "IPv6" in the "Protocol" column (See the first screenshot).

What I then did was that I simply went to edit the gateway, and the "Protocol" field had autoselected IPv4, so all I had to do was to click the save button. Now the gateway configuration went to say "IPv4" in the "Protocol" column, the "Status" column now said "Online" in green, and the "Name" column had "(active)" (See the second screenshot).

Voila! Now when I go to do a ping test, traffic goes out perfectly fine, DNS worked and updates check out!

Other than my redundant setup, I have two other OPNsense firewalls which are, rather average router configurations, one being my home firewall, and the upgrade to 19.7 and 19.7.3 did not produce this problem on either of those.

So in summary, it would appear that, somehow, when 19.7 booted up and looked at the configuration file, it somehow managed to interpret the gateway as being IPv6. As can be seen from the screenshot, the gateway was seemingly disabled. This seems rather odd, because it was working on 19.1.10, so why it was deemed to be defunct, and, according to the release notes added "in disabled state" makes no sense to me.

[mimugmail]

Nobody known why the gateway code in 19.1 even worked. And for introducing the gateway priorities it was completely rewritten so the code maintainer knows how it works. There may be scenarios where it can't work from code perspective, but worked from user perspective (which is now not working anymore).