Messages

[zfs_load="YES"currdev="zfs:OMEGA:"vfs.root.mountfrom="zfs:OMEGA"# This is the directory where modules (zfs.ko, etc) livekern.kmoddir="/boot/kernel"# This is the name of the binary INSIDE the 'kernel' directorykernel="kernel.opn"bootfile="kernel.opn"root_rw_mount="YES"]

I have one storage device ada0 which does include several partitions. The laptop is a Lenovo T480. This is not my main laptop but my "testing lab". I have one efi partition on ada0p1 which has got Limine installed. The linux systems using this partition as the boot partition. The ada0p2 is my BSDEFI, dedicated only for BSD based distros. FreeBSD and OPNsense. My FreeBSD zfs pool called ALPHA is living on ada0p10, size is 99GB. I have 118GB free space on ada0 starting at 4376576 and i want the opnsense's pool placing there. My goal is to install OPNsense on ada0p9 on zfs with 20GB size, using poolname OMEGA. I want make ensure that those pools are identical and independent and able to boot without interfering with each others.
The custom install for opnsense went well, however after reboot Limine not able the recognize for which entry belongs to which pool. I use GPT partition table and my BIOS had been set UEFI boot only.

This is what i was doing:

Code Select Expand

gpart add -t freebsd-zfs -b 4376576 -s 20G -i 2 -l OPNSENSE ada0


I was asking several AIs (grok, gemini, chatgpt, deepseek) and they were claiming that the opnsense and the freebsd zfs version might be different and that might causing the boot issue, however later on i was realizing not that is the core of the issue, but i was following the detailed command pointing to the zfs version to create the pool. I was trying this on my FreeBSD and also trying to create the pool on the OPNsense USB installation media as well, in shell, but both method was resulting boot issue after the installation finished and was trying the boot entries.
The command is:

Code Select Expand

sudo zpool create -f -o compatibility=openzfs-2.1-linux \
-o altroot=/mnt \
-O compression=off \
-O mountpoint=legacy \
OMEGA /dev/ada0p9

Code Select Expand

sudo mount -t zfs OMEGA /mnt

Then i was creating some "skeleton" dirs which are neccessary for booting up opnsense.

Code Select Expand

sudo mkdir -p /mnt/dev /mnt/proc /mnt/tmp /mnt/conf /mnt/mnt /mnt/var/run /mnt/root /mnt/home /mnt/usr/local/etc

Then setting up the permissions for tmp folder.

Code Select Expand

sudo chmod 1777 /mnt/tmp


This stops OPNsense from acting like a Live USB

Code Select Expand

echo "default" | sudo tee /mnt/etc/platform
echo "default" | sudo tee /mnt/usr/local/etc/platform


Creating fstab for OPNsense

Code Select Expand

sudo printf "OMEGA\t\t/\t\tzfs\trw\t0\t0\n" | sudo tee /mnt/etc/fstab



Next, i was creating the mntusb folder for the installation USB media of OPNsense.I was mounting the opnsense da1p4 data partition at /mntusb

Code Select Expand

sudo mkdir /mntusb
sudo mount /dev/da1p4 /mntusb
Then i was copy the USBs folders and their files to the pool.

Code Select Expand

foreach dir ( bin sbin lib libexec usr var etc root )
sudo cpdup -vv -I /mntusb/$dir /mnt/$dir
end

Next, create the kernel directory.

Code Select Expand

sudo mkdir -p /mnt/boot/kernel

The kernel file is living in /boot/kernel both on opnsense and on freebsd as well, but i was renaming th kernel for opnsense to kernel.opn as AI recommended it...But was not help. Anyway:

Code Select Expand

sudo cat /mntusb/boot/kernel/kernel | sudo tee /mnt/boot/kernel/kernel.opn > /dev/null
sudo cat /mntusb/boot/kernel/zfs.ko | sudo tee /mnt/boot/kernel/zfs.ko > /dev/null


Copy Boot Environment:

Code Select Expand

foreach bdir ( modules defaults lua zfs )
sudo cp -a /mntusb/boot/$bdir /mnt/boot/
end

Code Select Expand

sudo cp /mntusb/boot/device.hints /mntusb/boot/loader.efi /mntusb/boot/*.4th /mnt/boot/
sudo cp /mntusb/usr/local/etc/config.xml.sample /mnt/conf/config.xml

Write /mnt/boot/loader.conf:

Code Select Expand

sudo cat <<EOF > /mnt/boot/loader.conf
loader_brand="opnsense"
loader_logo="hourglass"
loader_menu_title="OPNsense Multiboot"
zfs_load="YES"
currdev="zfs:OMEGA:"
vfs.root.mountfrom="zfs:OMEGA"
kernel="kernel.opn"
bootfile="kernel.opn"
kern.kmoddir="/boot/kernel"
root_rw_mount="YES"
beastie_disable="YES"
autoboot_delay="3"
carp_load="YES"
pf_load="YES"
if_bridge_load="YES"
if_vlan_load="YES"
EOF
I did use the

Code Select Expand

sudo mount -t msdosfs /dev/ada0p2 /mnt/efi
sudo mkdir -p /mnt/efi/EFI/OPNsense
sudo cp /mnt/boot/loader.efi /mnt/efi/EFI/OPNsense/loader.efi

commands for the loader.efi file to copy it to the efi partition of ada0p2. This is only neccessary if i do the installation from the USBmedia.
On my FreeBSD the command is slightly different as the partition already mounted during boot on /boot/efi
The fstab for opnsense also has been modified adding the ada0p2 partition to /boot/efi later on during the installation process.

Then I was adding the "most important" lines according to AI:

Code Select Expand

echo 'currdev="zfs:OMEGA:"' | sudo tee /mnt/efi/EFI/OPNsense/loader.env
echo 'vfs.root.mountfrom="zfs:OMEGA"' | sudo tee -a /mnt/efi/EFI/OPNsense/loader.env

As AI said:
"You must place a file called loader.env on your EFI partition (ada0p2) inside the EFI/OPNsense/ folder. This is the only way to stop it from "defaulting" to FreeBSD." Bruh....it did not help anyway....

Next,was adding the boot partition to fstab

Code Select Expand

cat <<EOF >> /mnt/etc/fstab

/dev/ada0p2 /boot/efi msdosfs rw 0 0

EOF

The next step i did is the loader.conf file. I was adding the following:

zfs_load="YES"
currdev="zfs:OMEGA:"
vfs.root.mountfrom="zfs:OMEGA"
# This is the directory where modules (zfs.ko, etc) live
kern.kmoddir="/boot/kernel"
# This is the name of the binary INSIDE the 'kernel' directory
kernel="kernel.opn"
bootfile="kernel.opn"
root_rw_mount="YES"



My limine config looks like this:

/FreeBSD
comment: mymainsystem
protocol: efi_chainload
image_path: uuid(<partuuidofmyada0p2>):/EFI/FreeBSD/loader.efi

/OPNsense
comment: myfirewall
protocol: efi_chainload
image_path: uuid(<partuuidofmyada0p2>):/EFI/OPNsense/loader.efi


Additionally i did also create the loader.env file for FreeBSD as well and also was adding the
currdev="zfs:ALPHA:"
vfs.root.mountfrom="zfs:ALPHA"
lines for loader.conf.

About bootfs:
I did try several methods
1. set the bootfs only for ALPHA
2. set the bootfs only for OMEGA
3. set the bootfs both of them
4. set the bootfs none of them

Also was trying to delete the zpool.cache file on FreeBSD. But did not help.
FreeBSD requires a `zpool.cache` file to find the pool during the early boot stage. I was also trying it to generate the file but also did not help.
# Generate cache file for OMEGA
zpool set cachefile=/tmp/zpool.cache OMEGA
# Copy it to the destination
mkdir -p /mnt/boot/zfs
cp /tmp/zpool.cache /mnt/boot/zfs/zpool.cache


The following errors occured during the boot:

devmatch: Can't read linker hints file. devmatch: Can't read linker hints file.

devmatch: Can't read linker hints file.

devmatch: Can't read linker hints file. devmatch: Can't read linker hints file.

lumbtfu: lumbt_fu_read: open: /usr/local/share/lumbt-firmware/

lumbtfu: main: Firmware download failed!

devmatch: Can't read linker hints file.

devmatch: Can't read linker hints file.

Cannot 'start' webcand. Set webcand_enable to YES in /etc/rc.co Cannot 'start' webcand. Set webcand enable to YES in /etc/rc.com

devmatch: Can't read linker hints file.

add host 127.0.0.1: gateway loß fib 8: route already in table

devmatch: Can't read linker hints file.

add host::1: gateway 100 fib 8: route already in table

add net fe80::: gateway::1

add net ff02::: gateway::1

add net::ffff:8.0.0.0: gateway::1

add net::0.0.0.0: gateway::1


Or this: (but this had fixed by creating the "skeleton" dirs)

tunefs: /: could not open special device

mount_unionfs: /boot: Invalid fstype: Invalid argument

mount_unionfs: /conf: Invalid fstype: Invalid argument

mount_unionfs: /etc: Invalid fstype: Invalid argument

mount_unionfs: /home: Invalid fstype: Invalid argument

mount_unionfs: /root: Invalid fstype: Invalid argument

mount_unionfs: /usr: Invalid fstype: Invalid argument

mount_unionfs: /var: Invalid fstype: Invalid argument

pwd_mkdb: /etc/pwd.db.tmp: Read-only file system

pwd_mkdb: /etc/pwd.db.tmp: Read-only file system

Setting hostuuid: ca282ckb-654d-11b2-b78c-ee154fbba1fc.

Setting hostid: 0xdd7b3d30.

eval: cannot create /etc/hostid: Read-only file system

/etc/rc.d/hostid_save: WARNING: could not store hostuuid in /etc/hostid.

eval: cannot create /etc/machine-id: Read-only file system

/etc/rc.d/hostid_save: WARNING: could not store hostuuid in /etc/machine-id.

>>>>> Invoking import script 'importer'

Press any key to start the configuration importer:

mkdir:/conf/backup: Read-only file system

mkdir:/conf/sshd: Read-only file system

chmod: /root: Read-only file system

mkdir: /var/etc: Read-only file system

override r------ root/wheel uarch for /var/run/1d-elf.so.hints?

Loading kernel...

Failed to load kernel 'kernel.opn'

can't load 'kernel.opn'

Type '?' for a list of commands, 'help' for more

OK show currdev

zfs:OMEGA:

OK Is /boot/kernel/kernel.opn /boot/kernel/kernel.opn

OK boot

Loading kernel...

Failed to load kernel 'kernel.opn

can't load 'kernel.opn'

OK unload

OK set currdev="zfs: OMEGA:"

OK boot

Loading kernel...

Failed to load kernel 'kernel.opn'

can't load 'kernel.opn'


Any idea how to make it bootable both systems?
Currently if i remove the whole OMEGA partition, simply booting into my linux system and delete the partition FreeBSD will be bootable again. Otherwise both BSD system are unable to boot.