Hi there,
We're seeing a problem across all of our OpnSense v23.x installations where OpenVPN Site-to-Site tunnels are in use.
The symptom:
When one side (site) of the tunnel drops due to either an internet connection going down or a scheduled task like a weekly reboot, and the tunnel is re-established, traffic no longer routes across the tunnel.
The workaround (fix):
- Go to VPN >> OpenVPN >> Clients at the "client" site and disable the client side of the tunnel.
- Go to VPN >> OpenVPN >> Servers at the "server" site and disable the server side of the tunnel (if possible).
- At each site, go to System >> Routes >> Status and search on "ovp" in the search box. Find the destination networks, if there are any. If there are none, no worries. If there are some, then note the NetIF number (for instance, "ovpns9", or "ovpncX") and add it to your search field at the top. All the routes you see there need to be removed (Garbage can icon on the right). Remember, the tunnel is down. Those routes should not exist, but they still do. They need to not exist.
- Go to VPN >> OpenVPN >> Servers on the server side of the tunnel and edit it. Add "(ovpnsX)" to the end of the name, replacing X with the number you found earlier. This will help you in the future. Uncheck the "Disabled" box. Save.
- Go to VPN >> OpenVPN >> Clients on the client side of the tunnel and edit it. Add "(ovpncX)" to the end of the name, replacing X with the number you found earlier. This will help you in the future. Uncheck the "Disabled" box. Save.
The tunnel should now re-establish, and new routes will be created.
In the future when one of these tunnels drop, you can use the ovpn number you documented in the label of the tunnel to more quickly find and delete the routes that are stale and left over from the previous tunnel connection.
We started noticing this at multiple sites after upgrading to v23.x, but it's possible it existed before then and we didn't catch what update caused it. But it seems to be universal, affecting multiple sites that have no connection to one another.
Could someone on the Dev team look into this, if no one has already?
I'm sure someone could write some script wizardry to do this. I haven't had time to put my programming hat on and do it as of yet.
Thanks, all!
It's likely you'll need the two patches created by AdSchellevis -- if my interpretation is correct and this is the same underlying issue affecting the CSOs
https://forum.opnsense.org/index.php?topic=35447.0 (https://forum.opnsense.org/index.php?topic=35447.0)
I've applied these patches to a few firewalls, but looking at them, they seem to only affect the UI, not the underlying code that may create or destroy routes when they are initiated or dropped. Am I being dense, or is this the case?
The issue I'm seeing is that nothing seems to be consistently destroying/deleting the routes when a tunnel drops, and then because there's already "a route" when the tunnel re-establishes, the route command can't do it's job. But the old routes are also stale/dead and don't work.