Need help changing startup behavior of a plugin

[insomniac2k2]

Hello,

New to the forums and I hope that I am posting in the correct place.  Below is the rundown of my present configuration and dilema:

I have been using zerotier successfully for 3-4 months now on a single router multi-WAN configuration.  My final step is to run a HA router configuration w/ zerotier.  As of present, i have HA working perfectly, but i cannot figure out how to get zerotier to work with it.

Here is what works:
Both HA nodes can connect to the zerotier network. If I IP them differently, I can simply change routes in my.zerotier to manualy select which router i want to route traffic to. Though this is not sufficient for HA standards. It must be able to recover on its own.

In order to do this seamlessly, I have created a CARP address and associated it to the zerotier interface.  This works!   BUT, it only works if I go into zerotier and restart the plugin (or disconnect and reconnect) after a fresh boot.  This tells me that I need to have the zerotier plugin start VERY last after boot. Long after the CARP VIP comes up. OR, at least have something that restarts the zerotier plugin sometime after a fresh boot.

If I am going about this all wrong, please feel free to pitch another direction. This is the best approach that I have come up with over that last few weeks of troubleshooting.

thanks!

[franco]

Hi,

Long story short: zerotier needs to be adjusted to act correctly under CARP situations. OpenVPN has a patch for this when you have it listen on a CARP address. We also built something similar for FRR plugin. So there is no easy way to resolve this.

AFAIK, the zerotier plugin author is MIA.


Cheers,
Franco

[insomniac2k2]

Thank you for this information. Although it is not what i wanted to hear

I expected that this was the case. It's too bad really, because i feel that zerotier is a great product and works phenomenal as a SDWAN once configured correctly. Presently, this will not sway me from rolling with a single router/multi-wan configuration. It would just be nice to run HA OPNSense routers as well.

I guess my only real concern is whether or not we will have a zerotier plugin in the future (If the author is no longer participating).

[franco]

Isn't Zerotier kind of declining as well? I don't use it, but I can tell that interest is slowly but steadily declining in the community for one reason or another.


Cheers,
Franco

[insomniac2k2]

It may be an unfortunate truth, but I can say that from personal experience, it seems that there may be a decline because people don't understand enough to make it work in a routed environment.  I can tell you that after trial and error, it works very well as a SDWAN solution, and its pretty much plug and play once its sorted.  Personally, i feel that if there was better understanding of zerotier, it would become very popular, very fast.

On that note, when and if i get some spare time, I may do a brain dump of everything I have learned and implemented. As it may be a good preservation piece to drive a bit more adoption and awareness

[mimugmail]

It's declining because of this:
https://www.zerotier.com/pricing/

ZeroTier’s software is open source and free to use for most purposes including personal use, internal use within a business or academic institution, and evaluation for uses that require commercial licensing.

This means it's not free if you partner with other companies.
Doesn't scale really well

[insomniac2k2]

I don't mean to sound insulting or anything, but I disagree.  Most companies (In my experience), would never open up their infrastructure (SDWAN,DMVPN, etc) to other companies. Partner or not.  It would be a controlled portal of sorts.  There is just too much liability there.

Unless I just do not understand their licensing as well as i think I do, it would be a perfectly viable solution for any company that would like to have a very easy multi-wan, SDWAN solution. This would scale for free up to 100 routers, and then shift to a paid solution. Which in my opinion, is still very reasonable.

Am I not understanding your point, or the present licensing model?

[chrisarzu]

@insomniac2k2 I'm precisely trying to do the same thing. I'm running 2 opnsense VMs in HA in my home network, each one of them running in a separate host of my xcp-ng cluster.
I'm able to get to each one of the zerotier interfaces on each firewall, but if would be ideal if I can get to the VIP instead. What did you end up doing?
Under the zerotier plugin details, the maintainer shows as dharrigan@gmail.com, is that the person that would be able to assist with this (do you happen to know Franco?)
I love opnsense by the way, and decided to go with it instead of the other one out there specifically because opnsense had this zerotier plugin, the other solution doesn't.

[mgiammarco]

Hi,
I resurrect this old thread because I am doing research and development on this topic too.
First I ask a question: it seems to me zerotier package is supported in OPNsense because it gets regular updates.
Then have you tried "prevent interface removal" option in zerotier interface?
After that I need zerotier in HA too.
I reached the point to put:
- one zerotier client in master, one in slave
- ospf routing that promotes routes to current master.

I see that you want to put a virtual ip... what is your use case?

Thanks,
Mario