Messages

Post the output here and I will have a look for you too!

If the output looks good I would do a filesystem check (fsck or ZFS scrub) and the Health Check that OPNsense provides to make sure there is no corruption left behind.

Shoot! So at ~21 hours I bumped the keyboard, "Esc" key got wedged under the monitor, which exited the memtest and rebooted.

So I ran SMART checks (all supported test passed) and swapped the machine back.

If you have recommendations for Linux-y (Ubuntu is my go-to Live distro) commands to do the ZFS checks you can recommend, I'll do that at some point. (Or whatever the health check thing you mentioned is. Or if I can run the commands from the running OPNsense system - attempting to avoid a "Live" BSD though, just from my own comfort level.)


Also, might not be relevant, but at some point overnight (~3am local time), my "temporary" firewall stopped passing traffic, lights on NIC were good but no DHCP received on WANs, not responding to anything on LAN (or any other VLANs). I suspect this was because I YOLO'd it and didn't install os-realtek-re, but I guess we may never know, as I installed the package while troubleshooting the outage (lots of alerts to my phone), rebooted, and went back to bed.

And maybe run memtest86+ from your favorite Linux Live ISO Boot Image on a USB Stick for at least 24 hours ?
If you see errors very shortly after starting it you can stop ofcourse!

I downloaded MemTest86+ directly and ran that via Ventoy - sure enough, bad stick of RAM. I wouldn't have thought of that on my own because from my past experience, I've not seen RAM failure cause these symptoms, so thank you for pointing me in the right direction there.

I've got it running - will let it go 24 hours, or until it stops/is "complete", whichever comes first, and then try to check S.M.A.R.T. status.

How about a nice smartctl -a /dev/<whatever> output ? :)

And maybe run memtest86+ from your favorite Linux Live ISO Boot Image on a USB Stick for at least 24 hours ?

Any tips on how to get smartctl to run? Apparently admins don't have sudo permissions, and running without sudo yields permissions errors. (Smartctl open device: /dev/ada0 failed: Permission denied)
Suppose I can do that from a live USB as well, if needed. Will definitely be a bit since I'll need to build a temporary box to cutover to in the meantime.

perhaps signs of a hardware problem. Perhaps you could tell what hardware is involved.

It's a generic mini-PC; if there's a spot to a hardware report or something that would be useful, I can post that, but I didn't find anything while poking through the webUI menus. It would be great if there were better logs in OPNsense to help troubleshoot the issues.

This machine is not as easy to disassemble and would require significant downtime to do so, hence the lesser detail on hardware specs than the other post. Based on the dashboard, looks like 8GB RAM, 100 GB SSD, load averages 0.60, 0.42, 0.30. Not sure what other hardware information is relevant without being able to see/have more detailed logs or something specific to troubleshoot/change.

Kernel panic screenshot from this morning, immediately prior to submitting via the Reporter.

Hello,
After a very chaotic upgrade on my HomeLab firewall, I took extra care with my HomeProd, and the process went entirely differently. No packages were removed, the DHCP server was not cut over, etc. I had to do all that manually, even though I'd done the pre-work to make it a smooth transition.

I've had a few times now, over the past few weeks, two events that seem unrelated but both keep occurring. One, is the firewall itself crashing. This is annoying, but not a huge issue since it automatically reboots. A couple weeks ago (I think - I didn't write down exactly when), it had one of these crashes, only it didn't bring up the Kea DHCP server when it restarted. I've had it on my list to investigate more, and then this morning, there was no crash (uptime of 9 days, so perhaps that's when the previous crash was), but Kea was stopped again. In both cases that Kea has been stopped, clicking the "Play" button on the "Services --> Kea DHCP --> Kea DHCPv4" page is all I've had to do to resolve the issue.

As I was getting ready to make this post, I decided to check for updates, and in the process of applying them, the firewall crashed again. Kea came up properly from the crash, but it's concerning I have two different critical components (the kernel and Kea) that can't seem to stay running.

Every time I have a crash, I submit the logs using the webUI reporter page, but never get any feedback whatsoever on it.

Attaching the Kea logs where the service stopped. Also attaching a screenshot of the latest kernel panic.(May take a 2nd post because apparently I've hit a space limit.)

Was on 26.7.2 prior to this morning's updates; now on 26.7.3.

AFAIK that's a very old RealTek NIC Chip used as Onboard NIC for a lot of devices and might be declared as EOL by the people who build the FreeBSD Kernel so you could check that and see if there is a solution available for those who want to use it anyway ?

Yes, it's old - that's why I'm using this machine as a firewall box rather than a general-purpose computer.
I suppose it's *possible* that the upstream devs went to the work of removing support for some RTL8111 flavors while leaving other RTL8111 flavors supported, but that seems implausible.

Also, that wouldn't (shouldn't) affect the proprietary driver package, which as noted, also doesn't work, even though Realtek says it's the same driver for both the working and non-working chipset flavors.

I have done some more testing and found that this NIC is just not working at all as of 26.1, nor 26.7. I did a clean install of 26.7 (not upgrades) and it's the same problem - the NIC just isn't powering up at all. It still works fine with 25.7.
The physical chip is labeled:
RTL8111E
B9C77G1
GB48B

The biggest issues I have, is that this doesn't seem to be announced in ANY of the changelogs I've found, but certainly not the one presented at upgrade time.

Saying "don't use Realtek" when it's a hardware thing you can't just swap out, and has worked for years, seems rather disingenuous. It might be great advice for new builds, but does nothing to help those that OPNsense effectively bricked their hardware if they want updates.

EDIT: for anyone tracking, I have now opened this issue:
https://github.com/opnsense/src/issues/307

download os-Realtek-re and realtek kmod packages from here, copy on a stick and install manually


http://pkg.opnsense.org/FreeBSD:14:amd64/26.1/MINT/26.1/latest/All/

Thank you for this link. I was able to download and install this package (also had to download and install the realtek-re-kmod package first, as "pkg add" complained about it as a prerequisite/dependency).

HOWEVER, there was no change. I even rebooted, but still can not get any link/activity light from either the switch or the NIC with OPNsense 26.1 installed. (on old_hardware)

Testing 26.1 with the same config on new_hardware, and it works just fine with both versions of the Realtek driver (the included driver and the proprietary driver).

After reinstalling 25.7 on the old_hardware (minus HDD), and loading up the same config, it works. I haven't even installed the os-realtek-re package yet.

What changed from 25.7 to 26.1 that would have COMPLETELY broken my Realtek driver in my firewall (old_hardware), but not other Realtek chips (new_hardware)? If I do a ctrl+f on this page, nothing shows up for "real":
https://docs.opnsense.org/releases/CE_26.1.html

So I'm at a loss.

In cased it's useful, I ran this:

Code Select Expand

# pciconf -lv | grep -B3 network
re0@pci0:3:0:0:    class=0x020000 rev=0x06 hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x1043 subdevice=0x8432
    vendor    = 'Realtek Semiconductor Co., Ltd.'
    device    = 'RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller'
    class      = network

• os-realtek-re not available at all; I still can't find it in either Packages or Plugins, even after another update/upgrade and re-check (26.1.7 now).

I fixed this. So, in addition to the Settings tab --> Community in the "Type" drop-down, there is also a checkbox in the upper right of the "Packages" tab, that says "Show community plugins". That's what I was missing.

I am hoping to do more installs/troubleshooting over the weekend, so should have more updates then.

The DVD image is really for DVD burning AFAIK so no wonder only Ventoy worked eventually :)

In the past, the only way I've been able to get a bootable flash drive of OPNsense (or pfSense, so I assume it's a BSD thing?) is writing the ISO directly to a UFD. The IMG files never worked on a UFD, neither directly nor with Ventoy once that became a thing. The only way I've got IMGs to boot in the past was to write them directly to internal media.


BUT, I will definitely accept any fault for not re-trying the recommended tactics this round. :-) Habits built from repeated troubleshooting die hard, I guess.

How do you fix the chicken-egg problem of "install packages before doing config" and "config needed to reach internet to install packages"?

Especially when one of those packages is a driver that makes the NIC work? At some point the included Realtek driver worked "enough" that I could use it to download the proprietary driver, but that seems to not be the case anymore for some hardware. (I intend to validate the NIC hardware is still good sometime in the next couple days.)

Fortunately, I have a HomeLab specifically for testing these sorts of things first. It's also entirely possible I misread or skimmed over something in one (or more) of the upgrade pop-ups that could have made this go a lot smoother.

Relevant things to know ahead of time:

• I'm a native Ubuntu user for nearly 20 years. I'm comfortable in a terminal, but not BSD-native.
• My HomeLab OPNsense install is router-on-a-stick layout; there's one physical NIC but everything is VLANs.
• The NIC hardware is some form of Realtek; I had the os-realtek-re package installed previously (not sure if it disabled on this upgrade or an older one).
• I use DDNS and WireGuard to VPN into this network.
• I have several DHCP reservations for the LAN-side VLAN.
• I have multi-WAN setup.
• There are two physical machines - the one I tried to upgrade, that's still broke, I'll call it "old_hardware", and a machine I pulled out of storage to at least try and get back up and running a bit ("new_hardware").

The firewall old_hardware appeared to just never come back up from the upgrade. No DHCP, no establishing the WireGuard tunnel, nothing. Plugging in a keyboard/monitor indicated that the interface wasn't coming up properly. When I reboot it, I can see it saying "re0 up" in the boot logs, but it doesn't actually come up (doesn't list WAN VLANs as having received DHCP IPs, they're just blank). In fact, it's so "down", that the switch it's plugged into, never even lights up the lights for that port. The new_hardware firewall plugged into this same port, with the same cable, works fine. I've never had an issue with the physical NIC on old_hardware until now - only driver/software issues, so I don't think that's what's happening this time, either.

I downloaded the amd64 DVD ISO and tried to put it on a flash drive to install on new_hardware, but neither dd- nor cat-created UFDs were able to boot; I had to drop the ISO into a Ventoy drive to be able to boot it at all. (26.1.6). (Yes, I validated the checksum and then unzipped it first.)

At one point I tried a USB Ethernet adapter (ASIX chipset) in the old_hardware machine, but it didn't seem to have the kernel module needed for it.

Getting the config.xml onto the new hardware for a Linux-native seems... much more difficult than it should be. I tried booting a Live Ubuntu instance (on the freshly-OPNsense-installed new_hardware machine) and importing the zroot, but it was just a handful of empty folders. I tried formatting a USB Flash Drive (UFD) with Ubuntu as MBR/vfat and the "b" partition type, but that didn't seem to work (couldn't get it to mount, neither with "mount" nor "mount_msdosfs"). Eventually, I wiped/formatted/filesystem'd the UFD using the OPNsense instance on new_hardware, and then sneakernet'd that over to an Ubuntu box, which was able to read, and place, the config.xml I needed; then sneakernet back, mount, and I had my config imported (after a reboot).

Upon booting this imported config on new_hardware, I still had several issues, such as:

• no DHCP working at all. The ISC package was removed, and Kea was not enabled.
• no DDNS. The package was removed.
• os-realtek-re not available at all; I still can't find it in either Packages or Plugins, even after another update/upgrade and re-check (26.1.7 now).
• no export/migration option for ISC-->Kea. Allegedly there's supposed to be a CSV export/import function, but I can't find it.

I'm half tempted to just pull the drive from new_hardware and drop it into old_hardware and see what happens. I'm also tempted to install a couple major versions older than current, import the config, migrate all the DHCP stuff over, and retry the upgrade.

I'm not sure the best course of action, but I've already spent 4 hours of troubleshooting and another 45 minutes of documenting of said troubleshooting tonight, so I'm looking for the "easiest" path forward to have a functional old_hardware system again. Note that transplanting the HDD from new_hardware now will NOT give me that, due to being stuck on ISC with no way to export/import to Kea, even though I did fix the DDNS issue.

Also going forward - is there a way to force packages to auto-reinstall after system upgrades, especially critical things like drivers or DDNS?

Thanks! I hadn't come across or tried that yet. That, and "Disable force gateway" looked promising - however, after trying all 4 combinations - no dice.

I *do* think I figured out my issue though - rule order. Somehow I ended up with an asterisk where there should have been a gateway group specified! I still don't know about the 2nd part - when both WANs are on the same subnet - and why that doesn't work at all. However, the majority of my use-case, this is fixed, since forcing the traffic out the correct interface, means that the interfaces talking to each other, are not on the same subnet anymore. I'm going to mark this "partially solved" since it's working enough for me, but I still don't know why the "same subnet" issues seems to exist, or what I might be doing wrong there.

Hello,

I realize this may be an edge case - I'm hoping I'm just missing a setting somewhere that can fix this. I've searched (Google, forums, Reddit/GitHub, etc.) and not found what I'm looking for - some false starts, but nothing that matches my situation closely enough.

The setup is depicted in the diagram, but I have 3 ISPs to my home, all fiber ONTs, that each get dropped to their own VLAN.

I have two fully-updated OPNsense firewalls - one for Production and one for Laboratory (HomeProd and HomeLab).

Both OPNsense firewalls have been setup using both the WireGuard Road Warrior and Multi-WAN tutorials. I have laptop clients that I may connect directly to the LAN of either firewall, and then VPN-client over to the other firewall's network.


I'm having what I believe are two separate problems, that are both manifested in this setup. The problems as I see them:
- Problem #1 is that HomeProd OPNsense is sending out HomeLab WireGuard client traffic on the wrong interface (Sending out the ISP_C instead of ISP_A, even when all 3 ISPs are "up".)
- Problem #2 is that OPNsense does not seem to accept a WireGuard VPN connection from an endpoint that's on the same subnet as the interface accepting it. So, for example, the HomeLab instance, when the ISP_C range is (for example) 70.0.0.5/24, then it cannot accept any connections from 70.0.0.6/24, because it's in the same subnet.

What this means is that connections from clients on the HomeProd LAN to the HomeLab WireGuard instance fail, because they're going out the wrong interface, and that interface is on the same subnet as the HomeLab WAN interface.

However, connections from clients on the HomeLab LAN that hit the HomeProd WAN (ISP_A) interface, work just fine, since it's a completely different ISP (and thus not sharing a subnet).

I further tested this by unplugging the ONTs for both ISP_A and ISP_B, forcing HomeProd to failover to ISP_C as its primary, and thus making both firewalls' only functioning WAN interfaces, on the same subnet. In this case, no client connections in either direction work.

However, if I disable ISP_C on HomeProd, then traffic (correctly) forced out the ISP_A interface, and connections to HomeLab WireGuard work properly (and connections in the other direction continue to work, too, of course).

Caveats:
- when attempting to connect, the corresponding "server" firewall will show the client briefly with a green checkmark, but it only ever gets the first handshake/heartbeat (set to 20 second intervals), and eventually falls off
- I do have one device that had an "established" link (an IoT camera), and it has somehow managed a still-working VPN client connection from HomeProd LAN to HomeLab WireGuard, throughout my experimentation, including a reboot of the HomeProd firewall.
- All connections to either Firewall WireGuard instance made "away" from home work perfectly fine. It's just when trying to go between the two that's problematic at this time.