Messages

Just a quick glance at a home user, with 10G backbone, but currently nearly pinning a core with only ~600Mbps file transfer.  This is a Intel i3-9100  Please @Zenarmor re-consider this decision.  I've promoted this product since inception, this is a big thorn I'm struggling with. 

It is a known fact that Unbound takes up DHCP names for dynamic leases only dynamically. If you create a reservation, it will take an Unbound restart to pick it up.

I do not exactly know how Unbound works with DHCP services other than ISC, like Kea or DNSmasq for dynamic leases, but I assume it works the same.

For me it's not a reservation, just normal dynamic leases.  The instance the prompted this post was a 3d printer that I powered on, but it happens for every newly leased device.  No, reservations or anything out of the ordinary.

I'm having a mildly frustrating issue, not sure when it started, but if there is a solution I'm ready to chase it. 

When a new device comes online, it gets it DHCP address and I can see it and it's host name under leases.  I can get to it via IP, but if I type it's host name it doesn't resolve.  A nslookup host.domain.com returns host not found, however if I restart the unbound service it's immediately available and resolves without issue.  It will eventually resolve on its own but it takes a long time, I've never waited it out fully to know how long. 

Is there a polling frequency that allows Unbound to pull in new DHCP leases?  Not sure exactly how these two talk, would have assumed any new DHCP lease would trigger something to add it to Unbound but that doesn't seem to happen. 

I do wish that OPNsense had better wifi support, it would be handy once in a while for things.

There's a decent list of wifi cards supported in freebSD 14.3.
I just not sure any would act as AP, they're all clients.

I guess you could wifi the LAN and WAN side of OPNsense, WAN side connects to the shared wifi AP, but you would still need another AP on LAN side for laptop/printer/etc AND fw to connect to. AP just needs to be layer-2, all the compute stuff can get dhcp from fw, etc. Voila, cable-less firewall.

I think that could work actually if opnsense can act as a Wi-Fi client in the wan it'd be simple enough to have a separate AP for the LAN.  Just have to figure out a compact device with a supported card.

[...]but this GL trvel router is flakey[...]

That's too bad. The GL.iNet devices are generally well-regarded by the OpenWRT folks, and it runs a modded OpenWRT from the factory. So suggesting "try OpenWRT" is kinda out the window.

I've never tried FreeBSD/OPNsense as a wi-fi client (much less AP), but I'd expect it to work OK with supported hardware. Getting that in a small form factor device might take some work. I grabbed a couple PCI-e devices (AR9380 "ath" and AX200 "iwf") off eBay to test... one of these days. How much money do you have (as time=money)?

Heh. For my wireless access at home I use an OpenWRT device broken down into two bridges (here I go with the bridges again...) where the wireless is on one with no IP assigned and a DHCP IP on the other for management only. The firewall is broken down to only separate the two bridges, and wireless clients are isolated. DHCP for both bridges is handled on my firewall. Similar concept to passeri's, but limited to my needs.

Yea that's what I thought too, I'm wondering if we just ended up with a bad unit.  Right out of the box I struggled just to get it to take updates, it'll show updating, reboot then be on the original firmware, usually after 4-5 retries/reboots it'll take the firmware.  I'm getting some interface errors in the logs, AI (if that can be trusted in the least) seems to think I have a hardware issue of some type or a firmware bug. 

Hate to even replace it, when it works it fits the bill perfectly. It's small, low power, quiet, and feature rich.  But I'm getting tired of getting nearly weekly calls of internet issues and constant pings from Uptime Kuma that it's down or high latency. 

Trying to figure out a better solution for my mother in law in an assisted living apartment complex.  They have shared WI-FI in the building but my mother in law has needs for smart TVs, printers, IoT etc. that needs to talk.  So for security and to allow her devices to talk I thought no big deal, grabbed a GL-MT3000 travel router, connected it to the apartment Wifi as WAN, setup her lan and wifi network/s, zero tier back to my opnsense firewall so she can access Jellyfin.  All is pretty good, but this GL trvel router is flakey, kinda reminds me of routers in the old days that needed a daily reboot to remain stable. I'm getting tired of fighting it...

Anyone have any thoughts on how I could plop and opnsense box down and use the Wi-Fi Wan?  I think I recall OPNsense doesn't like Wi-Fi adapters.  But thinking just rebuilding the "travel router" idea but with OPNsense instead of this goofy GL router. 

Secondly any other solutions that I might not be thinking of.   

I understand a company needs to make money. I am therefor happy to pay for my home subscription. If multicore support won't be in the free version, ok for me. Not nice, as multicore is plain standard nowadays as you other guys correctly stated, but ok.

It won't be ok however not to include multicore in home subscription. The upper plans are too pricey for my home purposes. I really think to quit home subscription, as I do not agree with that policy.

Think twice, if I were you, I would offer multicore for free, or include in home subscriptions whatever, to get more customers. Otherwise I guess you would lose them...

Locking multi-core support and limiting the number of policies in the home licensing is really gimping that tier IMO.  Making the tool cost prohibitive at a home license level. 

Just look how much Oracle charges per core... ROFLMAO...

I think you are confusing multi-core support with per-core licensing. Per-core licensing cost is pretty common across the enterprise stack. 

Oracle ZFS docs apply to Oracle hardware (former Sun) only, neither Linux nor FreeBSD.

FreeBSD uses GPT partitions. As do Ubuntu and Debian, if you follow the guide by zfsbootmenu.org. If you follow the OpenZFS guide for Ubuntu, it's /dev/disk/by-id.

FreeNAS and TrueNAS use GUUIDs. It's complicated.

Now the fact that the FreeBSD handbook is so outdated it is blatantly wrong - does not match in any way what the FreeBSD installer will (correctly) do - needs to be addressed. I'll poke some people.

Confusing is the big key word here, I'm a slight noob with FreeBSD and a full on noob with ZFS I've just been lost this whole time.

Regardless I appreciate the help and guidance here.

All good. I'd be interested into those guides you mentioned. If there is misleading documentation out there, we ought to do something about that.

https://sotechdesign.com.au/how-to-add-a-drive-to-a-zfs-mirror/

https://docs.oracle.com/cd/E53394_01/html/E54801/gayrd.html

https://askubuntu.com/questions/1301828/extend-existing-single-disk-zfs-with-a-mirror-without-formating-the-existing-hdd

https://www.devroom.io/2024/03/07/zfs-upgrade-single-disk-to-mirror/

I can go on and on, the issue is I didn't look up a freebsd specific guide. Figured ZFS was ZFS and a mirror is a mirror.  Ultimately I didn't know what I didn't know, so I guess my search was flawed.  But everything seemed to mostly agree, so figured that was the right path, and in my head a mirror is well... a mirror. I think in raid controller, on an old school RAID card I'd add in my disk, add it to the mirror, let it resilver/sync, go about my life and never think of it again.

Edit: I understand why this wouldn't have worked now, with a RAID controller, the OS and UEFI bootloader just points at the card instead of disks directly unlike ZFS.  I know know that boot info would have to exist on both disks in this instance.  IF it was just for storage I assume all that wouldn't be needed and the above guides would have been accurate.   

ada1p1, yes. Sorry.

ok, copies are done.  Waiting on resilver to finish, estimated to be about an hour. 

Appreciate the detailed guidance. If you have a way to accept it, I'd be happy to buy you a beer or coffee/tea depending on your elixir of choice for your trouble. 

Is the source wrong here dd if=/dev/da1p1 of=/dev/ada0p1 bs=1m?  I get "dd: /dev/da1p1: No such file or directory".   Assuming source should be /dev/ada1p1 but don't want to assume again.

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:08:05 with 0 errors on Fri May 23 17:33:06 2025
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada1p4    ONLINE       0     0     0

errors: No known data errors


Code Select Expand

gpart show
=>       40  468877232  ada1  GPT  (224G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  451563520     4  freebsd-zfs  (215G)
  468875264       2008        - free -  (1.0M)

Weird the device name changed again, now it's ada1p4 vs ada0p4 when it was broken.  I'm so confused by that naming.

OK. This is the state that I wanted and with which I intended to guide you through the recovery process step by step. Now that your zpool is a bit messed up let's fix that first.

I assume

Code Select Expand

zpool status
results in "ada1p4" and "ada0" as the mirror disks?

We need to remove that ada0:

Code Select Expand

zpool detach zroot ada0


Now we take a breath and grab a coffee ... about those device names ...

FreeBSD enumerates the devices by some "hardware order" inherent in the drive, the PCIe bus, whatnot. Starting with 0.

So initially you had ada0 and ada1. Fine. Then ada0 failed. You removed it and rebooted. With only a single drive now present what was formerly ada1 is now ada0. It starts at 0. Always.

Then you inserted a factory new drive in the "first" (whatever that means) hardware position. After another boot that one is now "first" and becomes ada0 and what was initially ada1, then ada0, is now ada1 again.

FreeBSD just counts.


Now the boot process. For a PC system to be able to boot there needs to be a partition table and either - depending on the system - legacy ("BIOS") or EFI boot code in a matching partiton. When you install stock FreeBSD you can pick which to install. OPNsense installs both, just so not to bother the user with questions they cannot answer and always be able to boot, even if you replace your hardware and move your drive from e.g. a legacy system to am EFI system.

You can see that in your "gpart show" output. An EFI partition followed by a freebsd-boot (legacy) partition. Followed by swap and ZFS. ZFS must go into a partition of type freebsd-zfs, never to the whole disk.

You need the "boot thingies" on both disks, because you want to be able to boot of either of them in case one fails.


So now if that removal of ada0 succeeded first we create a partition table. The easiest way in case of identical drives is to copy it from the good one to the new one:

Code Select Expand

gpart backup ada1 | gpart restore ada0

Should the "new" drive not be entirely new and the above command fail because gpart does check if there is a partition table present, already, you can add the "-F" flag to that "gpart restore" command. It's just a reasonable safety measure. But since your new drive never had a partition table it should go well without "-F".

You can then check with

Code Select Expand

gpart show
that now both drives are partitioned the same.


Now that we have a ZFS partition to keep or zpool data we can attach that to the mirror:

Code Select Expand

zpool attach zroot ada1p4 ada0p4

Didn't it appear odd to have "ada1p4" but just "ada0" without a partitin when you did it the first time?

Anyway the zpool should now be resilvering and be done in no time as you can check with

Code Select Expand

zpool status
again.

Good? Next step, copy that boot code.


We copy both the EFI and the legacy partitions from ada1 to their respective counterparts on ada0:

Code Select Expand

# copy EFI boot
dd if=/dev/da1p1 of=/dev/ada0p1 bs=1m

# copy legacy boot
dd if=/dev/da1p2 of=/dev/ada0p2 bs=1m


That's it. Grab a beer. You have a redundant bootable system again. If you want redundant swap, too, which I recommend, we can do that in another round after your system is healthy again.


Kind regards,
Patrick

ok, resilvering again, hopefully correctly this time.

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat May 24 10:50:06 2025
        137G / 137G scanned, 2.20G / 137G issued at 119M/s
        2.24G resilvered, 1.61% done, 00:19:21 to go
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p4  ONLINE       0     0     0
            ada0p4  ONLINE       0     0     0  (resilvering)

errors: No known data errors

Once that finish (assume I should wait till resilver has finished) I'll copy the boot stuff.

Nooooo!

That's exactly how not to do it!

Your system will not be able to boot, when ada1 fails.

Why can't you guys wait for me to react to the post with the info I asked for. Good grief!

Ugh, sorry, EVERY guide I've read showed that was the process to rebuild. 

I"m disappointed in ZFS, thought it was going to be a huge value add.  But I'm really thinking about taking all my stuff back to using a classic raid controller.  ZFS documentation is poor and confusing and covered in trip wires and mines.   

So this seemed to work:

Code Select Expand

zpool attach zroot ada1p4 ada0

Shows resilvering now, but the naming looks weird:

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat May 24 09:55:56 2025
        137G / 137G scanned, 3.30G / 137G issued at 113M/s
        3.34G resilvered, 2.41% done, 00:20:13 to go
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p4  ONLINE       0     0     0
            ada0    ONLINE       0     0     0  (resilvering)

errors: No known data errors