On Sunday, October 18, 2026, my production OPNsense firewall failed to boot following an overnight automatic update from **26.7.6_3 to 26.7.7**, followed by a scheduled reboot.
The outage took down my household network and required several hours of troubleshooting using a local KVM console.
I eventually recovered the system by manually loading ZFS support through the FreeBSD bootloader. Subsequent investigation revealed that **32 files under `/boot` were zero bytes, all sharing exactly the same modification and inode-change timestamp: October 18 at 03:27:16 PDT.**
That timestamp is just 23 seconds after the package log recorded the OPNsense core upgrade.
I'm documenting the findings in one post because I intend to reinstall rather than continue troubleshooting the damaged installation. Hopefully this provides enough information for the developers to investigate.
## System information
| Item | Details |
|---|---|
| Previous OPNsense core | **26.7.6_3** |
| Current OPNsense core | **26.7.7** |
| Previous opnsense-update utility | 26.7.6 |
| Current opnsense-update utility | 26.7.7 |
| Architecture | amd64 |
| Commit | f1a83f72b |
| Filesystem | ZFS |
| Root dataset | zroot/ROOT/default |
| Boot mode | Apparently Legacy BIOS; hardware supports UEFI |
| Update mirror | mirror.wdc1.us.leaseweb.net |
| Repositories | OPNsense (priority 11), SunnyValley (priority 7) |
| Update date | October 18, 2026 |
| Local timezone | PDT (UTC-7) |
The firewall runs on physical x86-64 hardware with local console access.
The installed FreeBSD base/kernel version before the update was not independently established.
## 1. What happened
The automated firmware update ran at approximately 03:26 AM PDT.
The firewall subsequently rebooted around 04:00 AM, at which point it failed to start normally.
Symptoms observed:
- The normal OPNsense/FreeBSD boot selection menu did not appear.
- The console presented a minimal FreeBSD bootstrap loader interface.
- The kernel could be selected, but the system could not mount its ZFS root filesystem.
- ZFS support was not being loaded automatically, despite the relevant configuration being present.
- The firewall was inaccessible over the network until manual recovery.
The bootloader displayed:
```text
FreeBSD/x86 bootstrap loader, Revision 3.0
/boot/kernel/kernel
Hit [Enter] to boot immediately,
or any other key for command prompt.
```
The existing `/boot/loader.conf` included:
```text
loader_brand="opnsense"
loader_logo="hourglass"
autoboot_delay="3"
opensolaris_load="YES"
zfs_load="YES"
```
The configuration also contained the expected ZFS-related settings, including `vfs.zfs.vdev.min_auto_ashift=12`.
Despite this, the automatic boot process failed to initialize ZFS correctly.
## 2. Manual recovery
I accessed the firewall using a local KVM console and entered the FreeBSD loader environment.
After troubleshooting the loader commands, I was able to manually load the required ZFS functionality and boot the installed operating system.
Once booted:
- OPNsense 26.7.7 started successfully.
- LAN connectivity returned.
- The web administration interface became accessible.
- Root SSH access was enabled for investigation.
- The configuration was backed up to a separate workstation.
The precise successful bootloader command sequence was not preserved in the diagnostic notes, so I will not reconstruct it from memory.
**The important distinction is that the installed kernel and ZFS root filesystem were bootable when ZFS support was loaded manually.**
## 3. Discovery: 32 zero-byte files
The most significant finding was the condition of the Lua bootloader files.
The default loader is the Lua implementation, confirmed by identical inode numbers:
```text
261330 /boot/loader
261330 /boot/loader_lua
262023 /boot/loader_4th
```
The default `/boot/loader` and `/boot/loader_lua` are hard links to the same executable.
However, 16 essential Lua modules were zero bytes.
Running:
```sh
find /boot -xdev -type f -size 0 -print
```
identified **32 zero-byte files** across three directories.
### Lua bootloader files (16)
```text
/boot/lua/cli.lua
/boot/lua/color.lua
/boot/lua/config.lua
/boot/lua/core.lua
/boot/lua/drawer.lua
/boot/lua/gfx-beastie.lua
/boot/lua/gfx-beastiebw.lua
/boot/lua/gfx-fbsdbw.lua
/boot/lua/gfx-install.lua
/boot/lua/gfx-orb.lua
/boot/lua/gfx-orbbw.lua
/boot/lua/hook.lua
/boot/lua/loader.lua
/boot/lua/menu.lua
/boot/lua/password.lua
/boot/lua/screen.lua
```
Two OPNsense-specific Lua files remained intact:
```text
/boot/lua/brand-opnsense.lua 1875 bytes
/boot/lua/logo-hourglass.lua 2094 bytes
```
Both intact files were timestamped October 13.
### Boot console fonts (15)
```text
/boot/fonts/10x18.fnt.gz
/boot/fonts/10x20.fnt.gz
/boot/fonts/11x22.fnt.gz
/boot/fonts/12x24.fnt.gz
/boot/fonts/14x28.fnt.gz
/boot/fonts/16x32.fnt.gz
/boot/fonts/32x64.fnt.gz
/boot/fonts/6x12.fnt.gz
/boot/fonts/8x14.fnt.gz
/boot/fonts/8x14v.fnt.gz
/boot/fonts/8x16.fnt.gz
/boot/fonts/8x16b.fnt.gz
/boot/fonts/8x16v.fnt.gz
/boot/fonts/INDEX.fonts
```
The scan also returned one additional font, `8x16b.fnt.gz`, already included above. The complete set is fourteen compressed fonts plus the font index.
### Firmware file (1)
```text
/boot/firmware/iwm7265Dfw
```
## 4. Identical modification timestamps
Every one of the 32 zero-byte files had the exact same modification timestamp:
**2026-10-18 03:27:16 PDT**
I verified both modification and inode-change times using:
```sh
stat -f '%Sm %Sc %z %N' \
-t '%Y-%m-%d %H:%M:%S' \
/boot/lua/loader.lua \
/boot/fonts/INDEX.fonts \
/boot/firmware/iwm7265Dfw
```
Output:
```text
2026-10-18 03:27:16 2026-10-18 03:27:16 0 /boot/lua/loader.lua
2026-10-18 03:27:16 2026-10-18 03:27:16 0 /boot/fonts/INDEX.fonts
2026-10-18 03:27:16 2026-10-18 03:27:16 0 /boot/firmware/iwm7265Dfw
```
The complete directory listing confirmed the same timestamp for all affected Lua files, and the full timestamp check confirmed the same pattern for the affected font and firmware files.
This is a very unusual pattern.
The files span three separate directories but share the same timestamp to the second.
The Lua files also had permissions resembling:
```text
-r--r----- 1 root wheel 0 Oct 18 03:27
```
I have not established whether these permissions were altered by the same operation.
## 5. Firmware update timeline
From `/var/log/pkg/pkg_20261018.log`:
```text
2026-10-18 03:26:43 krb5-1.22.2_3 installed
2026-10-18 03:26:43 cyrus-sasl-gssapi-2.1.28 installed
2026-10-18 03:26:43 openldap26-client reinstalled: 2.6.15_1 -> 2.6.15_1
2026-10-18 03:26:43 opnsense-update upgraded: 26.7.6 -> 26.7.7
2026-10-18 03:26:53 opnsense upgraded: 26.7.6_3 -> 26.7.7
```
Then, at **03:27:16**, all 32 affected files show identical modification and inode-change timestamps.
The package log records the upgrade events but does not identify the operation that modified these boot files.
The synchronization is compelling, but I cannot conclusively attribute the truncation to a particular package script or base-system extraction operation.
## 6. Filesystem checks
After recovery, the ZFS root filesystem mounted successfully.
```text
Filesystem Size Used Avail Capacity Mounted on
zroot/ROOT/default 422G 108G 314G 26% /
```
The pool reported ONLINE with no known data errors during investigation.
The filesystem had approximately 314 GB available, so this does not appear to be caused by disk-space exhaustion.
There were no available ZFS snapshots:
```text
# zfs list -t snapshot -o name,creation -s creation
no datasets available
```
I did not perform a complete operating-system file-integrity audit, so I cannot rule out additional affected files elsewhere.
ZFS checksum integrity also cannot distinguish an erroneous application-level file truncation from an intentional write.
## 7. Bootloader and update-script inspection
I checked:
- `/boot/loader.conf`
- `/boot/loader.rc`
- `/boot/defaults/loader.conf`
- Installed Lua, Forth, and EFI loader binaries
- `/boot/lua/` contents
- `/usr/local/sbin/opnsense-update`
- Available package and update logs
The Forth `/boot/loader.rc` contained the expected initialization sequence, but the default loader is Lua-based, so the Forth startup script is not the relevant initialization path.
The Lua loader files were empty, including `loader.lua`, `config.lua`, `core.lua`, and `menu.lua`.
This is consistent with the missing menu and the failure to process the ZFS module-loading configuration.
I also inspected `/usr/local/sbin/opnsense-update`.
The utility contains an `install_base()` function using an extraction command of the form:
```sh
tar -C / -xpf ${WORKDIR}/${BASESET}
```
That operation can write files into the boot directories.
**However, I did not establish that `install_base()` executed during the October 18 update.**
An existing `.update.log` contained an earlier October 11 base/kernel update transaction. It did not establish that the same operation ran on October 18.
I found no direct references to the affected boot paths when searching `/usr/local/opnsense/scripts`.
## 8. Current operational state
The firewall is currently running OPNsense 26.7.7 following manual recovery.
I have:
- Restored household network connectivity.
- Confirmed web administration access.
- Established root SSH access.
- Exported and saved a configuration backup to another workstation.
- **Disabled automatic firmware updates/patching.**
- **Disabled scheduled weekly automatic reboots.**
I have deliberately not attempted to replace the damaged files or perform another firmware update.
The system is operational, but I do not consider it reliably rebootable.
My plan is to perform a clean installation using official OPNsense media, preferably with UEFI boot, and restore the configuration backup.
I am not comfortable trusting the integrity of the existing installation after finding 32 unexpectedly empty boot files.
## 9. Questions for the OPNsense developers
1. **Is there a known failure mode in the 26.7 update mechanism that could truncate multiple FreeBSD base-system files to zero bytes?**
2. Could a failed archive extraction, synchronization operation, or post-update script explain 32 files across three directories sharing the exact same timestamp?
3. Is there a known interaction between the 26.7.6_3 → 26.7.7 upgrade and the FreeBSD 15.x bootloader files?
4. Does this pattern suggest that other files may have been affected outside `/boot`?
5. Is there anything in the October 18 package changes that would specifically explain these files being modified immediately after the core upgrade?
I recognize that correlation with the update timestamp does not conclusively establish the responsible process.
Nevertheless, **32 boot-related files reduced to zero bytes, all sharing the same timestamp 23 seconds after the recorded package upgrade, followed by a boot failure**, seems significant enough to warrant investigation.
I am posting this primarily as an incident report for the maintainers and other users.
I do not intend to spend additional time troubleshooting the existing installation or participating in an extended diagnostic exchange. The firewall is essential to my household, and the outage already consumed several hours of a Sunday morning.
I've appreciated OPNsense's flexibility, particularly its advanced DNS and security capabilities, but this incident was extremely disappointing.
I hope the technical evidence is useful in identifying the failure mechanism and preventing similar outages for other users.