How do you handle drive failures in OPNsense?

[BigFreddy]

Hi everyone,

I was recently thinking about drive storage and how it is used in OPNsense, how often the drives should be changed and how much data OPNsense writes to it which I don't have a clear answer on. What comes with this topic is drive storage failures and my question is how do you guys handle that part of the firewall?

What's your philosophy or approach to drives used in OPNsense? Do you monitor drives for failures, if so, how and if not, how do you make sure that the failure of the drive doesn't cause any issues or opens up the firewall on the WAN side upon the failure?

Let's discuss about this topic and everything that comes with it.
Thanks

[Patrick M. Hausen]

Do you monitor drives for failures, if so, how

I use Scrutiny - and I am a bit disappointed I got zero feedback for my detailed instructions which I wrote up here:

https://forum.opnsense.org/index.php?topic=48101.msg242617#msg242617

I'd recommend to do something like this for every installation. Similar to watching temperatures e.g. via SNMP.

how do you make sure that the failure of the drive doesn't cause any issues or opens up the firewall on the WAN side upon the failure?

That scenario is so unlikely I would call it impossible. The firewall rules are loaded into the kernel. The default rule without any rule present is "deny all". So with all user processes crashing but the kernel just continuing to forward packets, your policy does not change. And should the firewall reboot but not be able to configure itself because it cannot read the configuration from the drive, all doors are shut.

[Mpegger]

I think for the home, if you're not making constant changes and tend to just "set it and forget it", just having a backup of the configuration and a recent ISO of OPNsense is enough to mitigate any sudden drive loss. A 2 disk mirror is also an option if you can swap drives when one suddenly dies.

[Seimus]

I lately started to use Beszel, which is slowly replacing my needs for the Grafana stack.
The last update added as well ZFS pool monitoring.

It works on every platform even OPN(FBSD), Proxmox PVE, LXCs etc.
So I have one uniform pane of view for metrics.

Beszel has as well a PR open for a possibility to monitor a device via SNMP, when that happens I can as well put it into my mikrotik switch.

Regards,
S.

[Patrick M. Hausen]

I lately started to use Beszel, which is slowly replacing my needs for the Grafana stack.

Interesting. Thanks!

[Seimus]

You are welcome,

Its really nice, and seamless to integrate. Basically I have all my LCXs, and whole network (expect mikrotik) in there.

Pic Of OPNsense ZFS pool monitoring.


Regards,
S.

[BigFreddy]

You are welcome,

Its really nice, and seamless to integrate. Basically I have all my LCXs, and whole network (expect mikrotik) in there.

Pic Of OPNsense ZFS pool monitoring.


Regards,
S.

This looks quite useful, mind sharing any links or insights how you setup and built the dashboard? I presume that dashboard is hosted on a spare device with some sort of agent installed on OPNsense?

[Patrick M. Hausen]

The dashboard is preconfigured. Yes, it also needs a separate Linux/Docker device. And an agent on OPNsense.

Start here:

https://www.beszel.dev/

[Seimus]

Yop as Patrick said, the dashboard for all monitored systems is prereconfigured and standardized.
You have some options how to show it like cards or pages etc. But its standardized.

Beszels works in a HUB&Spoke model.
Where the HUB is the server that collects & visualizes metrics.
Where the Spoke is the server/system you collect from.

Beszel spoke can run as a binary (recommended) or in a docker container on the monitored system.
I run the HUB in an LXC as docker container and Spoke beszel clients are binaries.

All you need, is to install somewhere the HUB, in the HUB add new system and choose platform + deployment method.
Than you just copy over the command and execute it on the spoke.
In case of a binary its just command in case of a docker it will give you a compose or docker run command.

And yes, read the beszel docs, beszel automatically detects most of the stuff. If you however have multiple disks you need to add them extra for monitoring and SMARTs.

Regards,
S.

[yourfriendarmando]

Aside from the monitoring tools, there is the scheduler, which can be instructed to run SMART short and long tests, on maybe weekly and monthly intervals, respectively.

In the event of drive failure. I have the most recent backup of the config.

On external disks, I carry a bootable Linux distro, and use it to deploy images to servers and firewalls. In the case of Opnsense, I have images created from pre-configured VMs, meant to cover 80% of most needs for a new site.

I use dd to the block device. after it's done, I resize the main partition, mount the EFI partition, drop the backup config in a directory.

Reboot to the deployed system, run:
zpool online -e nameOfPool /dev/adaXXX
to instruct ZFS to fill the remainder of the disk.

Make a copy of the config and do a search and replace of the interface device names from the virtualized to the real cards, like vtnet0 to igb0 for example. Copy the config over to /conf/config.xml, and immediately reboot, or with init 6 or your preferred method.

It's the quickest way to cover most disasters, of course the are more tools like HA (High Availability) to get a few firewalls running , with one of those acting as a spare to the other.

It doesn't hurt to have spare units with at least 4 ethernet ports, and spare SSDs on hand. I don't bother deploying images onto them until it's necessary.

The dd tool is fast enough, and the steps above can get you a working firewall within the hour.

It's easier to clone systems with a config and images kept up to date, it makes the hardware more easily replaceable when it fails.

cheers