Site to Site connection by using ZeroTier

[ismurdegus]

Hey guys
Hope everyone doing well.....

I can't fine much documentation online so I decide to ask the expert....

I have two office site connected on the internet via OPNSense firewall.
They both using a 4G modem, so no static internet IP.

Site1 -> 192.168.3.0/24   /  Firewall IP: 192.168.3.1

Site2 -> 192.168.2.0/24  /  Firewall IP: 192.168.2.1

Both sites have a printer and a NAS that we need share across location.

I successful install the ZeroTier plugin and join the ZeroTier network ID.
I set the two firewall with a static IP4 address inside the firewall interface setting....the same that is set on the ZeroTier web interface.

After that I don't know what to do because I can't ping any of address from one site to the other.

Do I missing something?

Thanks a lot

[mukky]

Hello guys,

I just want to share what i've been successfully done atfer
struggling couple days to solve it. 

I have 3 opnsense server node at 3 different cities.
Each of them are using Internet Service provider with private IP
a.k.a "behind CGNAT" (no Public ip address at all).
All of those 3 opnsense server using  >> "zerotier plugin" <<
to connect to zerotier central service.

For furter Reference, i did setup each of opnsense server base one
tutorial on youtube: https://www.youtube.com/live/Zp5vKPLAYdc?feature=shared

I did setup each of them as follows:

Opensense City 1 (Jakarta) -- Zerotier Assigned IP: 10.144.77.1
in this server i have two LAN Subnet:
-- Main Office: 192.168.1.0/24
-- Family: 192.168.2.0/24
-- Servers Farm:: 192.168.3.0/24

Opensense City 2 (Bali) -- Zerotier Assigned IP: 10.144.77.2
In this server i have three LAN Subnet:
-- Motel Room: 10.10.0.0/16
-- Bar: 10.20.0.0/16
-- Management Office: 10.30.0.0/24

Opensense City 3 (San Diego-CA) -- Zerotier Assigned IP:10.144.77.3
In this server I have two LAN Subnet:
-- Family: 192.168.99.0/24
-- Servers Farm: 192.168.88.0/24

Afther finished doing that tutorial i do have same problem with him,
sometimes connected couple minutes then droped then connect again.
Or sometimes it doesn't connected at all all day long. Also it have spikes
of the "Zerotier packets" on each of the server - i assumed it was
called >> "software laser issue" <<, to solve it i do couple things as follow:

At Zerotier Central web Application: >> Advanced>Managed-Routes << i put following
route configuration as follows:

192.168.1.0/24 via 10.144.77.1
192.168.2.0/24 via 10.144.77.1
192.168.3.0/24 via 10.144.77.1

10.10.0.0/16 via 10.144.77.2
10.20.0.0/16 via 10.144.77.2
10.30.0.0/24 via 10.144.77.2

192.168.99.0/24 via 10.144.77.3
192.168.88.0/24 Via 10.144.77.3

Then I put "local.conf" configuration code on every opnsense server
(Jakarta, Bali, San Diego-CA). This can be done via opnsense web gui administration
which is at  >> "VPN:Zerotier:Settings" << as follows:

{
  "physical": {
    "192.168.1.0/24": {
      "blacklist": true
    },
    "192.168.2.0/24": {
      "blacklist": true
    },
    "192.168.3.0/24": {
      "blacklist": true
    },
    "10.10.0.0/16": {
      "blacklist": true
    },
    "10.20.0.0/16": {
      "blacklist": true
     },
    "10.30.0.0/24": {
      "blacklist": true
     },
    "192.168.99.0/24": {
      "blacklist": true
     },
    "192.168.88.0/24": {
      "blacklist": true
    }
  }
}

Save & Apply !

It requires to restart every single opnsense server above
(Jakarta, Bali, and San Diego-CA) to work properly.

In result,
any of PC computer/laptop/phone (whithout installing zerotier on PCs/laptop/phone) that connected
to LAN network on one city could connect to any PC Computer/server on the two others cities
and vice versa. For example, i have laptop connected to LAN on opnsense server at jakarta
that have ip address: 192.168.1.7, it can connect file sharing on the NAS Server
on the opnsense San Diego-CA, simply connect to NAS Server local ip address 192.168.88.8.

If you would like to limit it, - based on your needs - you have to configure 2 things as follow:
1. Configure >> "Advanced>Managed Routes" << on Zerotier central web Application.
2. Configure firewalls rules at >> "Firewall:Rules:Ztier" << on every single opnsense server
connected to zerotier central.

I hope this can help others who have same difficulty to solve.

Regards,
Mukky Van Djava.