IPSec / OpenVPN Setup stopped working after upgrade to 23.7.10

[eduard.bicerimatei]

Hello all, ran myself into a corner during some maintenance work and need your help!
We have a 3x OPNSense VM Setup (IE, US and SA) in 3 separate AWS VPCs. Each VM has 2 ENIs (one public and one private). Each VM runs an OpenVPN server (public) and 2 IPSec tunnel to the remaining 2.

Before the upgrade, i was able to login with openvpn into the IE VM, and then access US resources (traffic was going local -> VPN -> IPSec -> VPC).
This morning i did the upgrade to 23.7, and after the final restart 2 things broke:
- the IE VM can't reach US resources (other VMs behind the US VM) -> i had to manually add static routes on the IE VM with specific US subnets going via ipsec
- me, on vpn, i can't reach US resources (even if the IE VM can reach now, because of the static routes).

I went over all settings in IPSec, checked and double checked the tunnel settings, everything looks ok.
Then I went over the Firewall rules, also looks ok.

Not sure where I should start debugging this (kinda newbie on the networking side) so any help would be greatly appreciated.

[eduard.bicerimatei]

Hello,
Some more context.
I was able to restore the "before upgrade" VM and noticed it has extra routes (netstat -4rn) that are not defined in the GUI.
So, I would suspect a config file somewhere that defines the routes and gets applied after restart, but it somehow got lost during the upgrade...

Any pointers in the right direction would be greatly appreciated.