Hi all,
It's possible this is covered somewhere and I missed it in my searches. If so, point me in the right direction and thanks in advance!
Under IPSec, there's a setting called Dead Peer Detection that would send an R_U_THERE packet every X seconds and if it didn't get a response, declare the tunnel dead and you could do resulting actions (Clear, Restart, Trap, etc.). It was far from perfect and there were good reasons not to use it in specific scenarios, but it existed as an option and could be helpful.
Under OpenVPN, there doesn't seem to be the same thing. Here's what I'm running into:
- The tunnel drops because one side or another has an internet issue. While I could go down the rabbit hole of why the ISP sucks, it happens and it's a fact of life.
- The issue is, when this happens, the firewall on the remote side (and sometimes the local side too) seem to retain routes to that network over the OpenVPN number. You can see this in System > Routes > Status and search on the ovpns# or ovpnc#, depending on which side of the tunnel you're looking at in that moment.
- With these stale routes in place, even if the tunnel re-establishes, the routes don't seem to work.
- The only solution that I've found that consistently works is to disable the tunnel on both sides, go and clear the routes out of the table above, then re-enable the server, and then re-enable the client.
Ideally I'd like to automate this process. Here's what I'm thinking, and if someone has a more elegant solution, I'm all ears:
I'm thinking all this could be done with Monit.
Set up a check in Monit for every minute, pinging a remote host. I'd suggest a remote host that isn't the remote firewall, because that way you ensure that the route is working all the way through to the end destination and not just to the remote firewall. So in my case, I use the NVRs at the remote site.
Assumptions:
- Monit is running on each firewall of the tunnel
- Each running once a minute
- Each sync'ed with time using NTP
On the 10.0.0.0/24 network side (server side):ping -4 -c 4 -S [LAN_Address] [Remote_NVR_IP]
(I've also seen in the forums where people do this with tcpdump. I'm not particular. I just need something that can be interpreted as a success/failure)
if [failure]
- disable the tunnel (what command? **See below) I was thinking psgrep server[##] but it seems psgrep isn't an option in OPNSense, and ps -ax | grep server[##] returns both the real process ID and the grep command process ID)
Discovered that 'pkill -f server[##]' returns the process id, which can be killed off, but that doesn't disable the tunnel. It just kills the current one.
- delete all the routes for that tunnel from the routes table (what command?)
- enable the tunnel (what command?)
On the 192.168.72.0/24 network side (client side):ping -4 -c 4 -S [LAN_Address] [Remote_NVR_IP]
if [failure]
- disable the tunnel (what command?
- delete all the routes for that tunnel from the routes table (what command?)
- wait 15 seconds (sleep 15) - this gives the server side time to "reset"
- enable the tunnel (what command?)
Can the above actions be scripted? If so, does someone have a template for this already in Monit?
(Edits for clarification)