23.1.7_1 broke my Firewall (Fixed)

[My_Network]

Hi every one!,

I upgraded to the latest version on Opnsense *23.1.7_1* and iv’e not been able to come back online since. My firewall is between my modem and a Cisco router. It was working fine on 23.1.6 in conjunction with Zenarmor. But now the pingeer  that tests the reachability of my « far gateway’s » (10.10.0.0/24 | 10.10.10.0/24 | 10.10.20.0/24 | 192.168.1.0/24 VIA static routes that point to 192.168.12.0 /24 via 192.168.12.1 (Cisco WAN interface),  keeps bouncing around erratically from an online status to no ping at all but with the green online status. And they refuse to actualy reach the router like it was doing before. Has anybody experience this? I’m totaly in the dark here. Tried reinstalling from scrach but it does noting to help. It was all working fine with 21.1.6. In then mean time i reactivated my "Residential Gateway" and IP NAT OUSIDE on my Cisco router to bypass my Firewall temporarely... Im at a loss here.

Hoping for some Help

Thank you,

Nic

[franco]

Hi,

Can you make sure to update to 23.1.7_2 and then apply the following on top?

https://github.com/opnsense/core/issues/6544#issuecomment-1535790249


Cheers,
Franco

[My_Network]

Hi Franco,

Just did the upgrade to 23.1.7_2 with the proposed patch on github, and at first glaced it fixed the eractic pinger behaviour. I will get back to you later when i move back my network behing my firewall.

Thank you,

Nic

[franco]

Thanks, in that case I'll weave that into the hotfix.


Cheers,
Franco

[My_Network]

Hi Franco,

So as promised here were I stand with version 23.1.7_3. Attached you will find a screen capture of "single gateways" currently running with all updated packages but with a #opnsense-revert -r 23.1.6 in shell to force the firewall to use 23.1.6 instead of 23.1.7_3. VLAN'S 10-20-30-40 and CISCO_WAN_INT are all "FAR GATEWAYS" configured with a routes as shown in screen capture called "Routes". These networks all opperates on my cisco 891f running downstream via 192.168.12.0/24 (Router Wan interface). Has soon as I update to 23.1.7_1-2-3 and try to add a gateway or reboot the firewall it renderes my network to the ground.

Thank you,

Nic

[struppie]

Similar problems here. Did the update from 23.1.6 to 23.1.7_3 with no configuration change (details in attached screenshot). Before the update everything was working fine.

Scenario:
(Internet) ---- (public IP - Router - private IP Network A) ----- (private IP Network A - OPNSense - private IP Network B) ----- (private Network)

After the update I couldn't access from my private Network any public machine of the Internet.
No ping, no http etc.
ping to private IP Network A of Router worked, but ping to public IPs did not work.

Restored backup => Everything fine again.

Seems something got broken with the update...

[BiTRiP]

Same here, since 23.1.7_x suddenly starting to have weird behaviors while nothing is changed on config.
With updates to 23.1.7_2 and 23.1.7_3 I hoped this was fixed but unfortunately not yet.

From my OpenVPN tunnels is one tunnel not responding (ping) on tunnel-subnet anymore (from both sides) but local LAN and remote LAN are still working.

Other tunnel is not responding on tunnel subnet and remote subnet.

Rebooted both ends but no luck.

Restored back to 23.1.6 all working fine again. 

[franco]

Ok, so to bring a little structure into this:

1. Are you using default gateway switching? If yes does it sort of work if you disable default gateway switching?

Because that was the only thing being switched over...

2. How do your routing tables look if it works on 23.1.6 vs. 23.1.7(_3) where it doesn't work. Usually "nothing works" is easy to spot in terms of entries in the routing table... I sounds a bit like hoping it just works vs. explicit configuration that is going on here.


Cheers,
Franco

[My_Network]

Hi Franco,

No, default gateway switching is not enable since im not using the gateway for multiwan purpuse but for Far gateways. I will try to send you the info running on 23.1.7_3 without the usualy "Someone broke the internet again"..... situation.

Thank you,

Nicolas

[struppie]

Hi Franco,

1: No default gateway switching in use

2: I will try to do another update attempt and report on this

Thanks for taking care!

[struppie]

Hi Franco,

I did the update from 23.1.6 to 23.1.7_3 again.
Immediately after the update everything still runs fine.
Then I did a reboot and the issue was back, that no traffic to the public internet was working.

Your hint to look at the routing table was good.
The comparison of before and after the update+reboot shows, that the first entry of the routing table was missing!
The one with "destination" default etc - refer to attached screenshot (red rectangle).
Interestingly the status of the gateway was marked as "online".

After adding the default route as a static route as interims solution, everything is working fine again.
But I think the default route should be created automatically as in former times?

[franco]

It's rather odd that default routes are missing and default gateway switching is not in use.

What is your WAN setup? IPv4 DHCP?


Cheers,
Franco

[struppie]

I configured a WAN interface (using vlan) with static IPv4 (192.168.30.254).
Additionally I configured the IPv4 upstream Gateway on this interface, which is the single Gateway I have.
(inner leg of a second router to the public internet)

Looks like this:
Scenario:
(Internet) ---- (public IP - Router - private IP Network A) ----- (private IP Network A - OPNSense - private IP Network B) ----- (private Network)

OPNSense: (private IP Network A) is the 192.168.30.254
Router: (private IP Network A) is the gateway IP 192.168.30.1

[franco]

I have an idea... Can you grep for this:

# opnsense-log | grep refusing


Cheers,
Franco

[struppie]

seems to be a good idea

Here's the output:

Code: [Select]

root@OPNsense:~ # opnsense-log | grep refusing
<11>1 2023-05-10T11:43:30+02:00 OPNsense.dimo.nil opnsense 285 - [meta sequenceId="8"] /usr/local/etc/rc.bootup: ROUTING: refusing to set inet gateway on addressless wan
<11>1 2023-05-10T11:43:35+02:00 OPNsense.dimo.nil opnsense 17719 - [meta sequenceId="32"] /usr/local/etc/rc.routing_configure: ROUTING: refusing to set inet gateway on addressless wan