OpenVPN - Routing bug?

[anomaly0617]

Hey folks!

I'm not sure if we call this a bug or ... what.

Here's what I've discovered, and reproducing it seems easy enough.

On a given firewall for a multi-site (10+ sites) organization, I previously used IPSec for the site-to-site connections.

IPSec tunnels work, but they do not allow for multi-WAN failover. For instance, we've recently moved from using "Dedicated" circuits (Costly, slow) to using multiple carriers with "Best Effort" circuits (Inexpensive, and generally fast). So we have two ISPs coming into each building. If the primary connection fails, the secondary connection seamlessly takes over by utilizing Gateway Groups.

However, IPSec tunnels tend to rely heavily on IP addresses for connections and identifiers. There are no options for "Try this connection as your primary, then this one as your fallback, and then on the other end, try this connection as your primary, and this other connection as your fallback."

This is where OpenVPN shines. Admittedly you have to define one site as a "parent/server" site and the other as a "child/client" site, but the multiple connection points thing is much easier to do, because in the client portion of OpenVPN, I can say "connect here on this port first, and then here on this port second."

However--

The desire by management for RoadWarrior VPN connections such that "once you connect to one site, you are connected to all sites" has become a problem. In my OpenVPN RoadWarrior configuration, I can define "Remote Networks" and include all the subnets of all the other sites. Works like a charm, until it creates a massive problem.

We noticed this when one internet connection was behaving flaky, and we were failing over to the other internet connection. We would (seemingly randomly) have trouble with sites connecting to other sites. And after about 6 months of frustration, I think I've finally found what's happening.

I started putting the Client or Server interface in the name of the OpenVPN profile. So for instance, parent/servers were ovpns{1,2,3,4,5,...} and child/clients were ovpnc{1,2,3,4,5}. This corresponds to the netif field in System >> Routes >> Status.

Lets assume that the RoadWarrior VPN connection, which was created when I first commissioned the site, is "ovpns1". And then after that, depending on the site, we added "ovpns{2,3,4...}" or "ovpnc{1,2,3...}".

When a site would stop routing to another site, I'd have to go in, disable the OpenVPN on both sides, go to the routes table, clear any leftover routes to the destination network(s) on each side out, and then restart the server, then restart the client. And this is where it gets interesting...

For many sites, I migrated them back to IPSec for stability reasons, and if the internet connection is being flaky, I'm just deactivating one IPSec tunnel and activating the second one. Which I can do automatically using Monit. But today I went to a site where I still had the OpenVPN Server/Client setup for the site-to-site, and I noticed when I went into the routes table that the "RoadWarrior" OpenVPN Server (ovpns1) had routes to all the other networks still there, even though no one was connected to the RoadWarrior VPN. So I cleared out all the ovpns1 routes to other sites and restarted the OpenVPN tunnel. It's back alive again.

Here's my theory:
All is well until someone connects to the RoadWarrior VPN connection, at which point the routes for the RoadWarrior VPN are created and it disrupts the Site-to-Site routes. Then that person disconnects and the routes do not delete. Which would explain the unpredictability of when this happens.

Here's my proposed fix/solution:

• Routes need "weights" or "costs" assigned to them. The RoadWarrior routes need a "higher cost" or "lower weight" than the Site to Site VPN tunnel routes. And there needs to be a way to script out clearing routes, because I have not found that command in BSD yet.
• There needs to be a cleaner way to delete routes via scripts. I've figured out ways to start and stop OpenVPN and IPSec tunnels using Monit. It would be nice if there was a way to write a global script that clears routes after I stop a tunnel and before I start it back up again.