I am going to deploy another OPNsense fw at other remote site. I planned on using WireGuard, but it has been a hassle to get it working. My sites primary site-to-site was WireGuard, but it has been very unstable, so I am switching to IPsec route-based.
I got the IPsec route-based tunnel up, but I could not pass any traffic or even ping between the two OPNsense (main fw and remote fw). Currently, I am staging the remote fw for deployment and it is currently directly connected to my security zone. So the IP of my main fw is 10.0.5.1 and the remote is 10.0.5.108. The bogons and rfc1918 has been disabled on the WAN interface of the remote fw for now.
I SSH-in to the main fw and pinged the remote fw's ipsec interface (192.168.1.6) and got this:
92 bytes from 127.0.0.1: Time to live exceeded
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 0054 cb89 0 0000 01 01 0000 127.0.0.1 192.168.1.6
I checked the routing table in CLI and it shows this:
192.168.1.1 link#26 UHS lo0
192.168.1.2 link#26 UH ipsec1
192.168.1.5 link#27 UHS lo0
192.168.1.6 192.168.1.5 UGHS lo0
It doesn't make any sense. ipsec1 is my site-to-site to my other remote site (This one is working and has been deployed), but the second (192.168.1.5 & 192.168.1.6) showing as lo0. When I run a ping, the Live View shows that the Interface is Loopback. I also have the Live View from the remote fw attached. At least at the remote site, it seems like it is using the correct interface which is ipsec1. It seems like a ship in the night situation here. The tunnel is up why can't these two firewalls pass traffic?
Please see the attached Live View screenshots.
I think I fixed my issue with IPsec route-based VPN. It seems like the option for duplicating the Phase 1 is a bad idea. The behavior was it will established the tunnel between two firewalls, but traffic-related wise it is a ship in the night behavior. After deleting the duplicated Phase 1, and recreated it from scratch, and restarting FRR, fixed my issues with IPsec.
I wish the WireGuard works again.