ZeroTier as a replacement for IPSec: Your Opinions, Please?

[anomaly0617]

Hi all,

We maintain a number of multi-site locations where we've used IPSec and fully-meshed the networks (ie: every location can talk to every other location). The obvious issue with this is the upkeep. Doing the math, if I have 11 locations, that means that every firewall has 10 IPSec tunnels. (Locations * (Locations -1)). So, in the instance of 11 locations, I'm maintaining 110 individual tunnels.

And this is where cloud-meshed VPN solutions (SD-WAN) enters the show. Instead of having to maintain 110 tunnels, I could maintain 11, and the cloud-meshed VPN solution would handle the routing. Or, worst case, I can manage the routes at the centralized console. Point is, the routing becomes easier.

For some of my locations, this is a non-starter. Those locations deal with protected information and an SD-WAN solution leaves too high a possible risk for someone to silently add their own node onto the network and then have time and access to all of the information they can vacuum up without our notice. Yes, I know that these solutions often offer multi-factor authentication at the cloud level. But even multi-factor is being hacked these days.

And this brought us around to ZeroTier. One of my engineers was complaining about what a pain in the rear maintaining all the tunnels was, and how SD-WAN would be so much better a solution. So I decided to give it a try with ZeroTier.

I picked a location that is more or less a "test site" - no issues if that site gets hacked, there's nothing there for them to find. And for the other site (and eventually, sites), I picked a few of our staff's home networks that are in a similar situation - if they get hacked... well, that sucks, but they have firewalls and backups.

I followed this guide and tried to keep the configuration as simple as possible. Get it working reliably first, and then start building rules and filters as necessary. But I didn't get that far.

My experience was that after

• updating OPNsense to the latest version
• getting the zerotier plugin installed
• connecting it to my zerotier portal through VPNs >> ZeroTier and using the API I was given from the ZeroTier portal
• adding the site and approving it in the portal
• creating the network interface and assigning it the appropriate IP address from ZeroTier (note: /24, not /32!)

I would see an uptime of about 45 seconds to 2 minutes, after which my network monitoring software would go from "all green" to "all red" pinging the remote hosts.

I tried adding managed route(s) (example: network 192.168.92.0/24 routes through 10.147.17.197).
I tried rebooting firewalls.
My experience did not change.

I read through the ZeroTier manual. No luck.

I not only disabled the IPSec tunnels that were in place before, I deleted them out of a few firewalls and rebooted. No change.

So, I'm now curious if I did something very obviously wrong, or if this experience is relatable to others? I'm open to the idea of an SD-WAN solution if it's stable, but I'm not going to sacrifice stability for convenience.

Opinions/Comments welcome!