Recent posts

#1

German - Deutsch / Re: CrowdSec sperrt ständig de...

Last post by Patrick M. Hausen - Today at 08:11:44 AM

Prüf mal die Whitelists:

https://docs.crowdsec.net/u/getting_started/post_installation/whitelists/

#2

German - Deutsch / Re: CrowdSec sperrt ständig de...

Last post by JamesFrisch - Today at 07:53:03 AM

Die siehst du btw unter

https://deineOPNsense/ui/crowdsec/decisions/index

#3

German - Deutsch / Re: CrowdSec sperrt ständig de...

Last post by JamesFrisch - Today at 07:51:05 AM

Hallo, seit ein Paar Tagen sperrt CrowdSec unseren Terminalserver alle 2 bis 3 Tage aus. Ich muss ständig unter Decisions, die Sperre löschen.

Was triggert denn die Sperre?

Habe Alias mit der IP-Adresse IPv4 des Terminalservers erstellt, eine Regel unter Firewall\Rules\Floating erstellt.

Hmm... bin mir nicht ganz sicher, aber ich glaube die Automatic Rules greifen vor deinen manuellen Rules.
Sprich, du wirst von Crowdsec in den automatic rules geblockt, egal was du nachher für eigene rules erstellst.

Gestern nach der Erstellung der Regel habe unter Firewall Live gesehen, dass der TS grün war, heute leider rot. Was mache ich falsch? 

Gestern wurde er durchgelassen, heute blockiert. Die Blockade ist bei den meisten crowdsec scenarios nur 4h.
Schritt 1 wäre also mal rauszufinden, warum überhaupt die Sperre getriggert wurde.

#4

26.7 Series / Re: SSD drive died - I disable...

Last post by oc - Today at 05:41:14 AM

Despite not being "Real Intel SSDs" in my book they were seriously BULLETPROOF compared to the Samsung 850 Pro Series in the past so IMHO you can trust that SSD with your life :)

OK - I'll soon give it another try - as I wrote earlier, I'm now quite proficient in rebuilding the OPN :)

Whatever the status is : If you don't need it, DISABLE IT! ;)

Done - hostwatch is off :)

the mobile devices which tend to use random MACs
Force your Clients to Disable that nonsense...

Done :)

Thank you! :)

#5

26.7 Series / Re: SSD drive died - I disable...

Last post by oc - Today at 05:35:45 AM

I was going to ask about the environment. I do not use an IPS; I have most logging enabled on pf (all inbound, NAT and firewall only outbound). I have static IPs and a couple lightly-used servers, so ~500 sessions active at any given time (with some spikes if the data scrapers go nuts). OPNsense has written 2.44TB over ~600 days active, or about a day's worth on the SSD (according to SMART). Apparently ZFS compression works really well on OPNsense.

25TB/d kinda sounds like write-only storage territory. I like to keep logs for a fairly long time (currently "Maximum preserved files" = 200), but it would take some effort to read anything useful from a 5PB store. Is that 25TB distributed?

You generally don't need to do anything fancy to get (at least some of) the endurance benefits of overprovisioning - just don't fill your device. Explicit device-level overprovisioning will generally help write performance, though. I partitioned the SSDs in my old (current) primary server for 180GB on 256GB devices, just to be paranoid. This was in 2014, by the way. (This is why I generally look at endurance over 10 years, not 3 or 5.)

I wonder how consumer-type PSLC caches behave in a write-heavy environment (I haven't looked into it, as I don't have any).

The 25TB/day isn't in any way related to my opnsense :) - for that logging environment millions were spent, and it's still not enough. It's broken down in elastic shards, but still - it's hard to even ingest so much data, never mind to store and search it.

"I partitioned the SSDs in my old (current) primary server for 180GB on 256GB devices" - can you please enlighten me what I need to do to achieve that? I tried (at the most recent install) to manually configure the drive (which is what I'd do in Windows, for instance - my experience with unix filesystems is limited), but for some reason it installed but I ended with a non-ZFS filesystem (although I am virtually certain of having chosen ZFS - but I think there was a test opnsense install on it, and perhaps the installer didn't wipe it thoroughly - no clue). Thank you :)

#6

26.7 Series / Re: SSD drive died - I disable...

Last post by oc - Today at 05:29:50 AM

My apologies for the last post, I meant the Crucial MX500 and not the MX300 

You have TRIM enabled so unless you had a drive that didn't support TRIM, maybe it died because it was crappy?

Good discussion here: https://forums.servethehome.com/index.php?threads/can-a-1-dwpd-ssd-become-a-3-dwpd-just-with-overprovisioning.45641/

You can calculate Write Amplification Factor (WAF) from the MX500 SMART data. I would encourage you to get one of these, over-provision by 33% ~ 50% and watch the calculated WAF with your specific workload. As noted in the discussion linked to above, I run either Micron 5200/5300MAX or Crucial MX500 OP'ed by at least 33% for all my OPNsense, Ubiquiti, ESXi system drives etc. Even when the underlying OS didn't support TRIM (ubiquiti crap for example) I've seen these drives keep the WAF super low. You can usually get HPE branded Micron 5300 MAX drives for cheap. I use them for my OPNsense although it is tough to update HPE SSDs or even obtain firmware for them

Interestingly enough, amazon doesn't have any Crucial MX500 size in stock :( - I found one on newegg, shipped from Hong Kong, but the picture says 'Made in Mexico' - go wonder ;)

Yes, I start to believe that drive was a dud. And in hindsight I think I should have noticed something's off, as the dashboard starting being 'slow/sluggish' at times. With this drive (a crappy one, but new and 512GB, 1% capacity used) the opnsense dashboard is MUCH snappier with exactly the same hardware. Today I have installed CrystalDiskInfo on my other oldies computers, and drives that are ~15 years old (with plenty of unexpected hard reboots etc) are still in 60-70% health despite (one of them) having written 56TBs on it (I was surprised too!). And that's all on Windows 10, upgraded to 11. So not sure what TRIM Windows does...

#7

26.7 Series / Re: Need an option - if it isn...

Last post by Patrick M. Hausen - Today at 01:14:39 AM

Well despite the responses - I mean NAT without a 'pass' is pretty pointless -

I take issue with "Firewall rules are allowing traffic" no they aren't - the NAT bypasses firewall rules completely.

It isn't. Without pass the incoming packet is first put through destination NAT and then firewall rules apply. That's a perfectly fine way to implement granular control. Only issue is that your firewall rule needs to have the source address as is (no change here) and the destination address *after NAT*. What's the problem with that?

HTH,
Patrick

#8

26.7 Series / Re: Need an option - if it isn...

Last post by BenKenobi - October 19, 2026, 11:59:18 PM

Well despite the responses - I mean NAT without a 'pass' is pretty pointless -

I take issue with "Firewall rules are allowing traffic" no they aren't - the NAT bypasses firewall rules completely.

I found the solution here, it is still wrong IMHO that firewall rules are not enforced before NAT - should only be presented for NAT if it does not violate any block list which is exactly what the thread linked below does.

https://forum.opnsense.org/index.php?topic=46122.0

#9

26.7 Series / OPNsense 26.7.6_3 → 26.7.7: 32...

Last post by MuffMuff - October 19, 2026, 11:15:43 PM

On Sunday, October 18, 2026, my production OPNsense firewall failed to boot following an overnight automatic update from **26.7.6_3 to 26.7.7**, followed by a scheduled reboot.

The outage took down my household network and required several hours of troubleshooting using a local KVM console.

I eventually recovered the system by manually loading ZFS support through the FreeBSD bootloader. Subsequent investigation revealed that **32 files under `/boot` were zero bytes, all sharing exactly the same modification and inode-change timestamp: October 18 at 03:27:16 PDT.**

That timestamp is just 23 seconds after the package log recorded the OPNsense core upgrade.

I'm documenting the findings in one post because I intend to reinstall rather than continue troubleshooting the damaged installation. Hopefully this provides enough information for the developers to investigate.

## System information

| Item | Details |
|---|---|
| Previous OPNsense core | **26.7.6_3** |
| Current OPNsense core | **26.7.7** |
| Previous opnsense-update utility | 26.7.6 |
| Current opnsense-update utility | 26.7.7 |
| Architecture | amd64 |
| Commit | f1a83f72b |
| Filesystem | ZFS |
| Root dataset | zroot/ROOT/default |
| Boot mode | Apparently Legacy BIOS; hardware supports UEFI |
| Update mirror | mirror.wdc1.us.leaseweb.net |
| Repositories | OPNsense (priority 11), SunnyValley (priority 7) |
| Update date | October 18, 2026 |
| Local timezone | PDT (UTC-7) |

The firewall runs on physical x86-64 hardware with local console access.

The installed FreeBSD base/kernel version before the update was not independently established.

## 1. What happened

The automated firmware update ran at approximately 03:26 AM PDT.

The firewall subsequently rebooted around 04:00 AM, at which point it failed to start normally.

Symptoms observed:

- The normal OPNsense/FreeBSD boot selection menu did not appear.
- The console presented a minimal FreeBSD bootstrap loader interface.
- The kernel could be selected, but the system could not mount its ZFS root filesystem.
- ZFS support was not being loaded automatically, despite the relevant configuration being present.
- The firewall was inaccessible over the network until manual recovery.

The bootloader displayed:

```text
FreeBSD/x86 bootstrap loader, Revision 3.0
/boot/kernel/kernel
Hit [Enter] to boot immediately,
or any other key for command prompt.
```

The existing `/boot/loader.conf` included:

```text
loader_brand="opnsense"
loader_logo="hourglass"
autoboot_delay="3"

opensolaris_load="YES"
zfs_load="YES"
```

The configuration also contained the expected ZFS-related settings, including `vfs.zfs.vdev.min_auto_ashift=12`.

Despite this, the automatic boot process failed to initialize ZFS correctly.

## 2. Manual recovery

I accessed the firewall using a local KVM console and entered the FreeBSD loader environment.

After troubleshooting the loader commands, I was able to manually load the required ZFS functionality and boot the installed operating system.

Once booted:

- OPNsense 26.7.7 started successfully.
- LAN connectivity returned.
- The web administration interface became accessible.
- Root SSH access was enabled for investigation.
- The configuration was backed up to a separate workstation.

The precise successful bootloader command sequence was not preserved in the diagnostic notes, so I will not reconstruct it from memory.

**The important distinction is that the installed kernel and ZFS root filesystem were bootable when ZFS support was loaded manually.**

## 3. Discovery: 32 zero-byte files

The most significant finding was the condition of the Lua bootloader files.

The default loader is the Lua implementation, confirmed by identical inode numbers:

```text
261330 /boot/loader
261330 /boot/loader_lua
262023 /boot/loader_4th
```

The default `/boot/loader` and `/boot/loader_lua` are hard links to the same executable.

However, 16 essential Lua modules were zero bytes.

Running:

```sh
find /boot -xdev -type f -size 0 -print
```

identified **32 zero-byte files** across three directories.

### Lua bootloader files (16)

```text
/boot/lua/cli.lua
/boot/lua/color.lua
/boot/lua/config.lua
/boot/lua/core.lua
/boot/lua/drawer.lua
/boot/lua/gfx-beastie.lua
/boot/lua/gfx-beastiebw.lua
/boot/lua/gfx-fbsdbw.lua
/boot/lua/gfx-install.lua
/boot/lua/gfx-orb.lua
/boot/lua/gfx-orbbw.lua
/boot/lua/hook.lua
/boot/lua/loader.lua
/boot/lua/menu.lua
/boot/lua/password.lua
/boot/lua/screen.lua
```

Two OPNsense-specific Lua files remained intact:

```text
/boot/lua/brand-opnsense.lua   1875 bytes
/boot/lua/logo-hourglass.lua   2094 bytes
```

Both intact files were timestamped October 13.

### Boot console fonts (15)

```text
/boot/fonts/10x18.fnt.gz
/boot/fonts/10x20.fnt.gz
/boot/fonts/11x22.fnt.gz
/boot/fonts/12x24.fnt.gz
/boot/fonts/14x28.fnt.gz
/boot/fonts/16x32.fnt.gz
/boot/fonts/32x64.fnt.gz
/boot/fonts/6x12.fnt.gz
/boot/fonts/8x14.fnt.gz
/boot/fonts/8x14v.fnt.gz
/boot/fonts/8x16.fnt.gz
/boot/fonts/8x16b.fnt.gz
/boot/fonts/8x16v.fnt.gz
/boot/fonts/INDEX.fonts
```

The scan also returned one additional font, `8x16b.fnt.gz`, already included above. The complete set is fourteen compressed fonts plus the font index.

### Firmware file (1)

```text
/boot/firmware/iwm7265Dfw
```

## 4. Identical modification timestamps

Every one of the 32 zero-byte files had the exact same modification timestamp:

**2026-10-18 03:27:16 PDT**

I verified both modification and inode-change times using:

```sh
stat -f '%Sm %Sc %z %N' \
  -t '%Y-%m-%d %H:%M:%S' \
  /boot/lua/loader.lua \
  /boot/fonts/INDEX.fonts \
  /boot/firmware/iwm7265Dfw
```

Output:

```text
2026-10-18 03:27:16 2026-10-18 03:27:16 0 /boot/lua/loader.lua
2026-10-18 03:27:16 2026-10-18 03:27:16 0 /boot/fonts/INDEX.fonts
2026-10-18 03:27:16 2026-10-18 03:27:16 0 /boot/firmware/iwm7265Dfw
```

The complete directory listing confirmed the same timestamp for all affected Lua files, and the full timestamp check confirmed the same pattern for the affected font and firmware files.

This is a very unusual pattern.

The files span three separate directories but share the same timestamp to the second.

The Lua files also had permissions resembling:

```text
-r--r-----  1 root wheel 0 Oct 18 03:27
```

I have not established whether these permissions were altered by the same operation.

## 5. Firmware update timeline

From `/var/log/pkg/pkg_20261018.log`:

```text
2026-10-18 03:26:43 krb5-1.22.2_3 installed
2026-10-18 03:26:43 cyrus-sasl-gssapi-2.1.28 installed
2026-10-18 03:26:43 openldap26-client reinstalled: 2.6.15_1 -> 2.6.15_1
2026-10-18 03:26:43 opnsense-update upgraded: 26.7.6 -> 26.7.7
2026-10-18 03:26:53 opnsense upgraded: 26.7.6_3 -> 26.7.7
```

Then, at **03:27:16**, all 32 affected files show identical modification and inode-change timestamps.

The package log records the upgrade events but does not identify the operation that modified these boot files.

The synchronization is compelling, but I cannot conclusively attribute the truncation to a particular package script or base-system extraction operation.

## 6. Filesystem checks

After recovery, the ZFS root filesystem mounted successfully.

```text
Filesystem            Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default    422G    108G    314G    26%    /
```

The pool reported ONLINE with no known data errors during investigation.

The filesystem had approximately 314 GB available, so this does not appear to be caused by disk-space exhaustion.

There were no available ZFS snapshots:

```text
# zfs list -t snapshot -o name,creation -s creation
no datasets available
```

I did not perform a complete operating-system file-integrity audit, so I cannot rule out additional affected files elsewhere.

ZFS checksum integrity also cannot distinguish an erroneous application-level file truncation from an intentional write.

## 7. Bootloader and update-script inspection

I checked:

- `/boot/loader.conf`
- `/boot/loader.rc`
- `/boot/defaults/loader.conf`
- Installed Lua, Forth, and EFI loader binaries
- `/boot/lua/` contents
- `/usr/local/sbin/opnsense-update`
- Available package and update logs

The Forth `/boot/loader.rc` contained the expected initialization sequence, but the default loader is Lua-based, so the Forth startup script is not the relevant initialization path.

The Lua loader files were empty, including `loader.lua`, `config.lua`, `core.lua`, and `menu.lua`.

This is consistent with the missing menu and the failure to process the ZFS module-loading configuration.

I also inspected `/usr/local/sbin/opnsense-update`.

The utility contains an `install_base()` function using an extraction command of the form:

```sh
tar -C / -xpf ${WORKDIR}/${BASESET}
```

That operation can write files into the boot directories.

**However, I did not establish that `install_base()` executed during the October 18 update.**

An existing `.update.log` contained an earlier October 11 base/kernel update transaction. It did not establish that the same operation ran on October 18.

I found no direct references to the affected boot paths when searching `/usr/local/opnsense/scripts`.

## 8. Current operational state

The firewall is currently running OPNsense 26.7.7 following manual recovery.

I have:

- Restored household network connectivity.
- Confirmed web administration access.
- Established root SSH access.
- Exported and saved a configuration backup to another workstation.
- **Disabled automatic firmware updates/patching.**
- **Disabled scheduled weekly automatic reboots.**

I have deliberately not attempted to replace the damaged files or perform another firmware update.

The system is operational, but I do not consider it reliably rebootable.

My plan is to perform a clean installation using official OPNsense media, preferably with UEFI boot, and restore the configuration backup.

I am not comfortable trusting the integrity of the existing installation after finding 32 unexpectedly empty boot files.

## 9. Questions for the OPNsense developers

1. **Is there a known failure mode in the 26.7 update mechanism that could truncate multiple FreeBSD base-system files to zero bytes?**
2. Could a failed archive extraction, synchronization operation, or post-update script explain 32 files across three directories sharing the exact same timestamp?
3. Is there a known interaction between the 26.7.6_3 → 26.7.7 upgrade and the FreeBSD 15.x bootloader files?
4. Does this pattern suggest that other files may have been affected outside `/boot`?
5. Is there anything in the October 18 package changes that would specifically explain these files being modified immediately after the core upgrade?

I recognize that correlation with the update timestamp does not conclusively establish the responsible process.

Nevertheless, **32 boot-related files reduced to zero bytes, all sharing the same timestamp 23 seconds after the recorded package upgrade, followed by a boot failure**, seems significant enough to warrant investigation.

I am posting this primarily as an incident report for the maintainers and other users.

I do not intend to spend additional time troubleshooting the existing installation or participating in an extended diagnostic exchange. The firewall is essential to my household, and the outage already consumed several hours of a Sunday morning.

I've appreciated OPNsense's flexibility, particularly its advanced DNS and security capabilities, but this incident was extremely disappointing.

I hope the technical evidence is useful in identifying the failure mechanism and preventing similar outages for other users.

#10

General Discussion / Re: Restricted traffic on VLAN...

Last post by yourfriendarmando - October 19, 2026, 10:27:12 PM

Some ISPs, for example, Comcast/ Xfinity, and using their router box in bridge mode, very much dislike seeing altered frame priority set at the interface level. I learned that the hard way. If you're using your own cable modem, less issues.

I opted instead for simpler traffic shaping at the Layer 2 IP level for traffic that needs lesser priority.