[SOLVED] How can I automatically restart wg instance, if gw down?

[tessus]

Unfortunately none of the solutions here worked. The Renew DNS for Wireguard on stale connections cronjob doesn't work in my case, because wg reports the connection as active (not stale) even though the gateway is down. So the action that should be triggered to restart the wg service is not triggered.

My current workaround is rather hacky (hardcoded wg service id), but it works. I will look into the script that is triggered in the previous mentioned cronjob. When I first looked into it I saw that it determined the id automatically.

Without further ado...

Use pluginctl -S wireguard to find the id of the wireguard connection. Make a note of that id.
Check which sockets are in use for your GW monitoring: ls -la /var/run/dpinger_*.sock (Make a note of the one you want to monitor. e.g. /var/run/dpinger_INTERFACE_NAME.sock)

Create a file, e.g. /root/bin/check-wg.sh (permissions 0750) and replace the values for SOCKET and WIREGUARD_ID

Code Select Expand

#!/bin/sh

SOCKET=/var/run/dpinger_INTERFACE_NAME.sock
WIREGUARD_ID=ad78e2ad-7ed9-422d-a0b9-ee7ed2795a3

RES=$(cat $SOCKET |cut -d ' ' -f2-)
DT=$(date +"%Y-%m-%d %H:%M:%S %z")

if [[ "$RES" == "0 0 100" ]]; then
    echo -n "[$DT] "
    /usr/local/sbin/pluginctl -s wireguard restart $WIREGUARD_ID

    # If you don't use or want gotify notifications delete the following lines (except the fi)
    TOKEN="$(cat ~/.config/gotify/cli.json |jq -r .token)"
    URL="$(cat ~/.config/gotify/cli.json |jq -r .url)"
    TITLE="check-wg ($(hostname -s))"
    TEXT="Wireguard restarted"
    curl -m 10 -so /dev/null -X POST "${URL}/message?token=$TOKEN" -F "title=$TITLE" -F "message=$TEXT"
fi

Unfortunately you can't add a cronjob by using crontab -e, because any manually added entries will be removed (either after a reboot or firmware update - I didn't check which, but my entry wasn't persisted).

Thus create another file: /etc/cron.d/check-wg

Code Select Expand

SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
REQUESTS_CA_BUNDLE=/usr/local/etc/ssl/cert.pem

*/5 * * * * root /root/bin/check-wg.sh >>/var/log/check-wg.log

That's it.

[meyergru]

Your assessment:

Unfortunately none of the solutions here worked. The Renew DNS for Wireguard on stale connections cronjob doesn't work in my case, because wg reports the connection as active (not stale) even though the gateway is down. So the action that should be triggered to restart the wg service is not triggered.

is almost surely wrong. The way the cron script detects if a wireguard connection is stale is by looking at the last handshake age and see if it is too old (> 135s). That way, you can be sure that there is still an ongoing wireguard connection. It is beyond me how that handshake should occur with the gateway down.

You can check this yourself:

https://github.com/opnsense/core/blob/ade7e9e9c7887978abf3f425c57def324ebcac03/src/opnsense/scripts/wireguard/reresolve-dns.py

The command for testing is "/usr/bin/wg show all latest-handshakes" and the last column is compared against "date +%s". If the difference is > 135, the connection is restarted. Of course, this can take up to ~2 minutes and also, if the drop is caused by the remote side changing its IP and DNS caching gets in the way, for an even longer time, because multiple tries must be taken until the connection gets up again.

If I am wrong, please create a bug report on github.

[tessus]

Yep, I read the script, but it's not working. I had the cronjob (Renew DNS for Wireguard on stale connections) scheduled to run every 5 minutes and it happened twice that the gateway was down but wg2 was not restarted. Of course it happened while I was sleeping thus it was offline for hours. I always had to manually restart the wg2 instance.

Only after I created my script, I never had the problem (of manually having to restart the interface) again. (I get notifications when it happens - only one so far.)

Creating an issue is not that easy, because the gw is usually online. Only sometimes it goes offline and I have no idea why. So I cannot forsee when this will happen and I cannot afford to disable my script and encounter potentially hours of no VPN connection.
Maybe I can add something to my script so that it will only restart the interface automatically when I am sleeping. Although this is a hassle, because I don't wear a smartwatch or a health tracker, which means I have to set a status var manually every time I go to sleep and get up. Rather a no-no on my side.

P.S.: I still think the current architecture has it backwards. It should be possible to trigger a script, when a gateway is down. pinger supports this. There just needs to be a field in the GUI to specify the script to run. As I mentioned before I am not a GUI person, thus I don't know how to do that. Otherwise I would have created a PR already. It's 5 minutes work (if you know how to do it): one field in the UI and invoke the dpinger with an additional parameter.

P.P.S.: Please note that I am not saying that your script is not working. It is not working for me for some reason. I only added my solution in case somebody runs into the same issue and the cronjob doesn't work for them either.