[feature request] Add emergency reboot fallback

[szty0pa]

Since a few minor versions my routers (apu3 and Fujitsu Futro thin clients) are not able to reboot properly from the webgui or after an update. I suspect this might be caused by some faulty suricata-netmap-wireguard interaction (that is giving me trouble sometimes).
The problem is that the shutdown script is able to stop enough services not to be able to log back in or continue to use the router but not enough to actually reboot the router itself. (If i am lucky i can log in to the webgui just to restart sshd to be able to log in through ssh; sshd does not accept my user/password after a failed reboot attempt for some reason unless restarted.)
Logging in through ssh and issuing a 'sudo reboot' successfully reboots the router every time (when it is possible to log in at all). So i would suggest adding a forceful 'reboot' after trying to gracefully shut down services in case it did not work as that would increase resiliency a lot.

[bartjsmit]

I suspect this might be caused by some faulty suricata-netmap-wireguard interaction (that is giving me trouble sometimes).

I would concentrate on fixing the problem :-)

Reduce your possible causes by removing bits (temporarily) until it works properly and investigate the last removal. Either that, or start with a clean slate and add features until it breaks.

Bart...

[franco]

> I would concentrate on fixing the problem :-)

This for now, otherwise we will get more of those and then blocking reboots will be more prominent. Remember, you are asking for a manual failsafe but a lot of expectations and automation will not trigger / know when to trigger it. Worst case on major upgrades you could kill the operating system killing the upgrade taking "too long".


Cheers,
Franco

[szty0pa]

Reduce your possible causes by removing bits (temporarily) until it works properly and investigate the last removal. Either that, or start with a clean slate and add features until it breaks.

Sure thing. What breaks is monitoring a wireguard interface with suricata. Taking suricata out the equation (at least on the wireguard interface) everything works as it should.

This for now, otherwise we will get more of those and then blocking reboots will be more prominent. Remember, you are asking for a manual failsafe but a lot of expectations and automation will not trigger / know when to trigger it. Worst case on major upgrades you could kill the operating system killing the upgrade taking "too long".

I wasn't thinking of such a sophisticated solution, only sticking a 'reboot' at the end of /usr/local/etc/rc.reboot just in case. Best case it wouldn't even be reached, worst case it would forcefully reboot the router if 'shutdown -or now' doesn't manage for some reason. No accidental reboot due to delay would be involved.

[edit:] it wouldn't help. '/usr/local/etc/rc.freebsd halt' is not able to kill suricata, that's why the whole shutdown process goes sideways. (Suricata on my boxes can only be killed by SIGKILL but not SIGTERM for some reason.)