Recent posts

#1

26.7 Series / Re: Need an option - if it isn...

Last post by Patrick M. Hausen - Today at 01:14:39 AM

Well despite the responses - I mean NAT without a 'pass' is pretty pointless -

I take issue with "Firewall rules are allowing traffic" no they aren't - the NAT bypasses firewall rules completely.

It isn't. Without pass the incoming packet is first put through destination NAT and then firewall rules apply. That's a perfectly fine way to implement granular control. Only issue is that your firewall rule needs to have the source address as is (no change here) and the destination address *after NAT*. What's the problem with that?

HTH,
Patrick

#2

26.7 Series / Re: Need an option - if it isn...

Last post by BenKenobi - October 19, 2026, 11:59:18 PM

Well despite the responses - I mean NAT without a 'pass' is pretty pointless -

I take issue with "Firewall rules are allowing traffic" no they aren't - the NAT bypasses firewall rules completely.

I found the solution here, it is still wrong IMHO that firewall rules are not enforced before NAT - should only be presented for NAT if it does not violate any block list which is exactly what the thread linked below does.

https://forum.opnsense.org/index.php?topic=46122.0

#3

26.7 Series / OPNsense 26.7.6_3 → 26.7.7: 32...

Last post by MuffMuff - October 19, 2026, 11:15:43 PM

On Sunday, October 18, 2026, my production OPNsense firewall failed to boot following an overnight automatic update from **26.7.6_3 to 26.7.7**, followed by a scheduled reboot.

The outage took down my household network and required several hours of troubleshooting using a local KVM console.

I eventually recovered the system by manually loading ZFS support through the FreeBSD bootloader. Subsequent investigation revealed that **32 files under `/boot` were zero bytes, all sharing exactly the same modification and inode-change timestamp: October 18 at 03:27:16 PDT.**

That timestamp is just 23 seconds after the package log recorded the OPNsense core upgrade.

I'm documenting the findings in one post because I intend to reinstall rather than continue troubleshooting the damaged installation. Hopefully this provides enough information for the developers to investigate.

## System information

| Item | Details |
|---|---|
| Previous OPNsense core | **26.7.6_3** |
| Current OPNsense core | **26.7.7** |
| Previous opnsense-update utility | 26.7.6 |
| Current opnsense-update utility | 26.7.7 |
| Architecture | amd64 |
| Commit | f1a83f72b |
| Filesystem | ZFS |
| Root dataset | zroot/ROOT/default |
| Boot mode | Apparently Legacy BIOS; hardware supports UEFI |
| Update mirror | mirror.wdc1.us.leaseweb.net |
| Repositories | OPNsense (priority 11), SunnyValley (priority 7) |
| Update date | October 18, 2026 |
| Local timezone | PDT (UTC-7) |

The firewall runs on physical x86-64 hardware with local console access.

The installed FreeBSD base/kernel version before the update was not independently established.

## 1. What happened

The automated firmware update ran at approximately 03:26 AM PDT.

The firewall subsequently rebooted around 04:00 AM, at which point it failed to start normally.

Symptoms observed:

- The normal OPNsense/FreeBSD boot selection menu did not appear.
- The console presented a minimal FreeBSD bootstrap loader interface.
- The kernel could be selected, but the system could not mount its ZFS root filesystem.
- ZFS support was not being loaded automatically, despite the relevant configuration being present.
- The firewall was inaccessible over the network until manual recovery.

The bootloader displayed:

```text
FreeBSD/x86 bootstrap loader, Revision 3.0
/boot/kernel/kernel
Hit [Enter] to boot immediately,
or any other key for command prompt.
```

The existing `/boot/loader.conf` included:

```text
loader_brand="opnsense"
loader_logo="hourglass"
autoboot_delay="3"

opensolaris_load="YES"
zfs_load="YES"
```

The configuration also contained the expected ZFS-related settings, including `vfs.zfs.vdev.min_auto_ashift=12`.

Despite this, the automatic boot process failed to initialize ZFS correctly.

## 2. Manual recovery

I accessed the firewall using a local KVM console and entered the FreeBSD loader environment.

After troubleshooting the loader commands, I was able to manually load the required ZFS functionality and boot the installed operating system.

Once booted:

- OPNsense 26.7.7 started successfully.
- LAN connectivity returned.
- The web administration interface became accessible.
- Root SSH access was enabled for investigation.
- The configuration was backed up to a separate workstation.

The precise successful bootloader command sequence was not preserved in the diagnostic notes, so I will not reconstruct it from memory.

**The important distinction is that the installed kernel and ZFS root filesystem were bootable when ZFS support was loaded manually.**

## 3. Discovery: 32 zero-byte files

The most significant finding was the condition of the Lua bootloader files.

The default loader is the Lua implementation, confirmed by identical inode numbers:

```text
261330 /boot/loader
261330 /boot/loader_lua
262023 /boot/loader_4th
```

The default `/boot/loader` and `/boot/loader_lua` are hard links to the same executable.

However, 16 essential Lua modules were zero bytes.

Running:

```sh
find /boot -xdev -type f -size 0 -print
```

identified **32 zero-byte files** across three directories.

### Lua bootloader files (16)

```text
/boot/lua/cli.lua
/boot/lua/color.lua
/boot/lua/config.lua
/boot/lua/core.lua
/boot/lua/drawer.lua
/boot/lua/gfx-beastie.lua
/boot/lua/gfx-beastiebw.lua
/boot/lua/gfx-fbsdbw.lua
/boot/lua/gfx-install.lua
/boot/lua/gfx-orb.lua
/boot/lua/gfx-orbbw.lua
/boot/lua/hook.lua
/boot/lua/loader.lua
/boot/lua/menu.lua
/boot/lua/password.lua
/boot/lua/screen.lua
```

Two OPNsense-specific Lua files remained intact:

```text
/boot/lua/brand-opnsense.lua   1875 bytes
/boot/lua/logo-hourglass.lua   2094 bytes
```

Both intact files were timestamped October 13.

### Boot console fonts (15)

```text
/boot/fonts/10x18.fnt.gz
/boot/fonts/10x20.fnt.gz
/boot/fonts/11x22.fnt.gz
/boot/fonts/12x24.fnt.gz
/boot/fonts/14x28.fnt.gz
/boot/fonts/16x32.fnt.gz
/boot/fonts/32x64.fnt.gz
/boot/fonts/6x12.fnt.gz
/boot/fonts/8x14.fnt.gz
/boot/fonts/8x14v.fnt.gz
/boot/fonts/8x16.fnt.gz
/boot/fonts/8x16b.fnt.gz
/boot/fonts/8x16v.fnt.gz
/boot/fonts/INDEX.fonts
```

The scan also returned one additional font, `8x16b.fnt.gz`, already included above. The complete set is fourteen compressed fonts plus the font index.

### Firmware file (1)

```text
/boot/firmware/iwm7265Dfw
```

## 4. Identical modification timestamps

Every one of the 32 zero-byte files had the exact same modification timestamp:

**2026-10-18 03:27:16 PDT**

I verified both modification and inode-change times using:

```sh
stat -f '%Sm %Sc %z %N' \
  -t '%Y-%m-%d %H:%M:%S' \
  /boot/lua/loader.lua \
  /boot/fonts/INDEX.fonts \
  /boot/firmware/iwm7265Dfw
```

Output:

```text
2026-10-18 03:27:16 2026-10-18 03:27:16 0 /boot/lua/loader.lua
2026-10-18 03:27:16 2026-10-18 03:27:16 0 /boot/fonts/INDEX.fonts
2026-10-18 03:27:16 2026-10-18 03:27:16 0 /boot/firmware/iwm7265Dfw
```

The complete directory listing confirmed the same timestamp for all affected Lua files, and the full timestamp check confirmed the same pattern for the affected font and firmware files.

This is a very unusual pattern.

The files span three separate directories but share the same timestamp to the second.

The Lua files also had permissions resembling:

```text
-r--r-----  1 root wheel 0 Oct 18 03:27
```

I have not established whether these permissions were altered by the same operation.

## 5. Firmware update timeline

From `/var/log/pkg/pkg_20261018.log`:

```text
2026-10-18 03:26:43 krb5-1.22.2_3 installed
2026-10-18 03:26:43 cyrus-sasl-gssapi-2.1.28 installed
2026-10-18 03:26:43 openldap26-client reinstalled: 2.6.15_1 -> 2.6.15_1
2026-10-18 03:26:43 opnsense-update upgraded: 26.7.6 -> 26.7.7
2026-10-18 03:26:53 opnsense upgraded: 26.7.6_3 -> 26.7.7
```

Then, at **03:27:16**, all 32 affected files show identical modification and inode-change timestamps.

The package log records the upgrade events but does not identify the operation that modified these boot files.

The synchronization is compelling, but I cannot conclusively attribute the truncation to a particular package script or base-system extraction operation.

## 6. Filesystem checks

After recovery, the ZFS root filesystem mounted successfully.

```text
Filesystem            Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default    422G    108G    314G    26%    /
```

The pool reported ONLINE with no known data errors during investigation.

The filesystem had approximately 314 GB available, so this does not appear to be caused by disk-space exhaustion.

There were no available ZFS snapshots:

```text
# zfs list -t snapshot -o name,creation -s creation
no datasets available
```

I did not perform a complete operating-system file-integrity audit, so I cannot rule out additional affected files elsewhere.

ZFS checksum integrity also cannot distinguish an erroneous application-level file truncation from an intentional write.

## 7. Bootloader and update-script inspection

I checked:

- `/boot/loader.conf`
- `/boot/loader.rc`
- `/boot/defaults/loader.conf`
- Installed Lua, Forth, and EFI loader binaries
- `/boot/lua/` contents
- `/usr/local/sbin/opnsense-update`
- Available package and update logs

The Forth `/boot/loader.rc` contained the expected initialization sequence, but the default loader is Lua-based, so the Forth startup script is not the relevant initialization path.

The Lua loader files were empty, including `loader.lua`, `config.lua`, `core.lua`, and `menu.lua`.

This is consistent with the missing menu and the failure to process the ZFS module-loading configuration.

I also inspected `/usr/local/sbin/opnsense-update`.

The utility contains an `install_base()` function using an extraction command of the form:

```sh
tar -C / -xpf ${WORKDIR}/${BASESET}
```

That operation can write files into the boot directories.

**However, I did not establish that `install_base()` executed during the October 18 update.**

An existing `.update.log` contained an earlier October 11 base/kernel update transaction. It did not establish that the same operation ran on October 18.

I found no direct references to the affected boot paths when searching `/usr/local/opnsense/scripts`.

## 8. Current operational state

The firewall is currently running OPNsense 26.7.7 following manual recovery.

I have:

- Restored household network connectivity.
- Confirmed web administration access.
- Established root SSH access.
- Exported and saved a configuration backup to another workstation.
- **Disabled automatic firmware updates/patching.**
- **Disabled scheduled weekly automatic reboots.**

I have deliberately not attempted to replace the damaged files or perform another firmware update.

The system is operational, but I do not consider it reliably rebootable.

My plan is to perform a clean installation using official OPNsense media, preferably with UEFI boot, and restore the configuration backup.

I am not comfortable trusting the integrity of the existing installation after finding 32 unexpectedly empty boot files.

## 9. Questions for the OPNsense developers

1. **Is there a known failure mode in the 26.7 update mechanism that could truncate multiple FreeBSD base-system files to zero bytes?**
2. Could a failed archive extraction, synchronization operation, or post-update script explain 32 files across three directories sharing the exact same timestamp?
3. Is there a known interaction between the 26.7.6_3 → 26.7.7 upgrade and the FreeBSD 15.x bootloader files?
4. Does this pattern suggest that other files may have been affected outside `/boot`?
5. Is there anything in the October 18 package changes that would specifically explain these files being modified immediately after the core upgrade?

I recognize that correlation with the update timestamp does not conclusively establish the responsible process.

Nevertheless, **32 boot-related files reduced to zero bytes, all sharing the same timestamp 23 seconds after the recorded package upgrade, followed by a boot failure**, seems significant enough to warrant investigation.

I am posting this primarily as an incident report for the maintainers and other users.

I do not intend to spend additional time troubleshooting the existing installation or participating in an extended diagnostic exchange. The firewall is essential to my household, and the outage already consumed several hours of a Sunday morning.

I've appreciated OPNsense's flexibility, particularly its advanced DNS and security capabilities, but this incident was extremely disappointing.

I hope the technical evidence is useful in identifying the failure mechanism and preventing similar outages for other users.

#4

General Discussion / Re: Restricted traffic on VLAN...

Last post by yourfriendarmando - October 19, 2026, 10:27:12 PM

Some ISPs, for example, Comcast/ Xfinity, and using their router box in bridge mode, very much dislike seeing altered frame priority set at the interface level. I learned that the hard way. If you're using your own cable modem, less issues.

I opted instead for simpler traffic shaping at the Layer 2 IP level for traffic that needs lesser priority.

#5

General Discussion / [SOLVED]Re: Restricted traffic...

Last post by Mattps - October 19, 2026, 09:06:16 PM

I know, I deliberately didn't go into details in case someone shouted out something "MTU size". Both VLANs were tagged by the way.

Anyway I've decide  to move this to a physical interface and see how that goes.

#6

General Discussion / Re: Restricted traffic on VLAN...

Last post by meyergru - October 19, 2026, 08:16:19 PM

You did neither tell your configuration nor the topology, so this is only guesswork:

1. Mixing up tagged and untagged (V)LANs on FreeBSD/OpnSense does not work too well. You should have either one or the other.
2. Some hardware does mix up VLANs as well, especially cheap switches. For example, if you cascade switches, an edge switch might short-circuit both VLANs, which can lead to all kinds of problems.

#7

General Discussion / Restricted traffic on VLAN int...

Last post by Mattps - October 19, 2026, 07:47:55 PM

Hi,

I have created two vlan interfaces on a physical interface, using the guide. The first VLAN interface I created works really well. However, the second VLAN interface I added is experiencing odd behavior. Traffic is passing through from the LAN side - I can ping/trace route/and open a Google page. But if I try and pull any larger amounts of data - speedcheck, Azure Portal login page, the connections time out (even thought the pings stay constant).

Any suggestions here? Both VLANs have the same (Best, 0) priority settings. I've tried elevating this to but it doesn't make a difference.

Cheers,
Matt

#8

26.7 Series / Re: Setting up WAN failover

Last post by viragomann - October 19, 2026, 07:16:26 PM

I rebooted the router. Now everything is working (beats me).

I guess, there were existing states for these connections you were trying to access. So following packets were forced to the gateway as well.

You can simply flush all states in Firewall: Diagnostics: States: Action or search for the particular connection in the state table and drop it.

#9

26.7 Series / Re: Setting up WAN failover

Last post by beneix - October 19, 2026, 07:04:33 PM

I rebooted the router. Now everything is working (beats me).

Next step is to take the plunge on migrating my legacy firewall rules...I think I will let that wait until I have my fibre broadband back, the hassle around this failover WAN setup has caused enough pain 😊.

#10

26.7 Series / Re: SSD drive died - I disable...

Last post by nero355 - October 19, 2026, 07:03:16 PM

I'd categorise the two mentioned products as "prosumer".

I have dozens of the Samsung 970 at home, friends' homes, at work, ... never lost one so far.

- Samsung 840 Pro/850 Pro/970 Pro/980 Pro
- WD "Black" Series
=
All "Gaming SSDs" in my book, sorry! :)

I have an 128GB Intel 520 Series, which appears to have enjoyed Industry-leading 5-year warranty and historically low failure/return rates when released (in 2012, so it's also not a spring chicken).

But I did run a S.M.A.R.T on it, and it reports 96% health and ~10,000 hours of uptime (which is a lot, but perhaps no big deal).

Despite not being "Real Intel SSDs" in my book they were seriously BULLETPROOF compared to the Samsung 850 Pro Series in the past so IMHO you can trust that SSD with your life :)

Is disabling hostwatch still a thing?

I think I have seen a topic or two where it was, but a very recent one made it look like the service is well behaving now...

Whatever the status is : If you don't need it, DISABLE IT! ;)

the mobile devices which tend to use random MACs

Force your Clients to Disable that nonsense...

Another solution is to get a Micron MX300 Crucial MX500 (and specifically that model) drive - even a 512GB one will do.

Then use Micron's Executive software to over-provision the drive by 50% i.e. it's a 256GB drive for all intents and purposes.
The drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM.
You should check if TRIM is enabled for your filesystem - are you using ZFS or UFS?

The Crucial MX500 Series is one of my favorites too :)

I also have the original and even better M500 960 GB model in use!
There was also the M500 DC variant which was even better Power Loss Protection-wise but I am not sure if there is a MX500 DC too...

"drive will now be able to take 3-4 DWPD (drive writes per day) even without TRIM" - does that mean that it can take 3-4 times its provisioned capacity (3-4x256GB)? Or am I misunderstanding?

I am guessing it can, especially if there is some kind of Garbage Collection feature in it's firmware : Those Intel 520 models were really good at it!

_{I can't remember if the MX500 has this feature... Too long ago for my memory at the moment... d0h!}

Good discussion here: https://forums.servethehome.com/index.php?threads/can-a-1-dwpd-ssd-become-a-3-dwpd-just-with-overprovisioning.45641/

I run either Micron 5200/5300MAX or Crucial MX500 OP'ed by at least 33% for all my OPNsense, Ubiquiti, ESXi system drives etc.

NICE! :)

Even when the underlying OS didn't support TRIM (ubiquiti crap for example)

Isn't that a matter of editing fstab correctly ?!

TRIM is most useful when the device is full.

I disagree : It's ALWAYS useful!

It's applied to keep the performance of the SSD at optimal level after all :)