OPN as a PXE boot server

[cookiemonster]

This mini tutorial is intended to setup a PXE boot server in OPNSense.
The clever parts I have used with permission are documented by Kraileth in his blog:
Links:
https://eerielinux.wordpress.com/2021/01/24/multi-os-pxe-booting-from-freebsd-12-introduction-pt-1/
https://eerielinux.wordpress.com/2021/01/30/multi-os-pxe-booting-from-freebsd-12-required-services-pt-2/
https://eerielinux.wordpress.com/2021/02/05/multi-os-pxe-booting-from-freebsd-12-pxe-menu-and-bsd-pt-3/
https://eerielinux.wordpress.com/2021/02/20/multi-os-pxe-booting-from-freebsd-12-linux-illumos-and-more-pt-4-2/
All I’ve done is adapted it to OPNSense and I made a couple of choices for my needs and infrastructure availability.

What we’ll have at the end of this tutorial is OPNSense acting as a PXE boot server that can be used for clients to network boot on a LAN interface. As an working example we’ll be serving a FreeBSD 12 boot option. I am using OPNsense 21.7.2_1-amd64. An important note is this is for BIOS boot, not UEFI.

The only additional dependency required is a plugin “os-tftp” found in the “default” mirror of OPNSense i.e. OPNSense package repository.

As a baseline I describe my setup. I don’t have VLANs and my flat network is in the 192.168.5.0/24 range. OPN has a lan IP of 192.168.5.1. I have a FreeBSD-based NAS with a reserved IP of 192.168.5.5 that has an available webserver running on port 8081. I will use this storage+internal webserver to serve the large isos. It is perfectly possible and simpler to host them in OPN but I have limited storage on my firewall appliance and I don’t want to run a webserver on it either.
I use DHCPv4 and not v6.
I use Unbound for name resolution.

Prerequisites:
A shell account that can elevate permissions with sudo on OPN.
A shell account that can elevate permissions with sudo on a local webserver.
Internet connection to download some necessary files. I use a separate computer and then transfer to the firewall, to avoid costly mistakes. Some will prefer to do it all on the firewall itself.
Some familiarity with basic shell commands and utilities like wget, nano, etc.

Now we start.
1. Install the tftp plugin in OPNSense
Go to System > Firmware > Plugins. Search for os-tftp. Click on the + sign at the end of the row to install it.
You will get a message “The root folder for transfering files is /usr/local/tftp.”
Once the WebUI refreshes you’ll have an entry in Services > TFTP > General where the service can be enabled or disabled.

On a shell connect to OPNSense and verify the path exists. For me it didn’t so I created it

Code: [Select]

$ sudo mkdir -p /usr/local/tftp
Then back on the WebUI I enabled the tftp service and entered my OPN LAN address 192.168.5.1 and save. Now we have a tftp server listening. To check:

Code: [Select]

$ sudo sockstat -4l
Password:
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS     
root          in.tftpd            48436 4       udp4   192.168.5.1:69                    *:*
I did root around the system to figure out how the plugin sets things up but I couldn’t find much. I was fumbling in the dark with this. I see it does not use inetd but I can’t yet find it’s config. I found an entry in etc/rc.d/ftpd but not much else. Nevertheless it is running and working fine for our purposes, and starts after reboot.
If someone finds where the control and config files are, let me know.

1.1 Create the pxe boot files being given by ftpd

Code: [Select]

sudo mkdir -p /usr/local/tftp/pxelinux.cfg
sudo nano /usr/local/tftp/pxelinux.cfg/default
Insert the following and save:

DEFAULT vesamenu.c32
PROMPT 0
MENU TITLE PXE Boot Menu (Main)

LABEL bsd-oses
   MENU LABEL BSD Operating Systems
   KERNEL vesamenu.c32
   APPEND pxelinux.cfg/bsd


Now for a test, from a linux or FreeBSD machine on the LAN do:

Code: [Select]

$ cd /tmp
$ tftp 192.168.5.1
tftp> get pxelinux.cfg/default
Received 190 bytes in 0.1 seconds
tftp> quit
$ rm default
That confirms our tftp daemon on OPN is serving the files correctly. I was expecting to need to open the port 69 but it seems the default pass rule from the LAN takes care of it.
You will need to adjust the firewall rules if you don’t have that permissive rule in place.

Now we need to create the referenced file “bsd”:

Code: [Select]

$ sudo nano /usr/local/tftp/pxelinux.cfg/bsdinsert this:

MENU TITLE PXE Boot Menu (BSD)

LABEL main-menu
   MENU LABEL Main Menu
   KERNEL vesamenu.c32
   APPEND pxelinux.cfg/default

This step 1.1 has created a default file and a bsd file, both inside a subdirectory under our tftp root.
Now we’re ready to get those files they reference.

2. Get the PXElinux files.
Ordinarily I do this on a separate machine and then transfer the files but on this ocassion I’ll do it directly on OPN.

Code: [Select]

$ cd /tmp
$ sudo pkg fetch -y syslinux
$ sudo mkdir -p /tmp/syslinux
$ sudo tar -C /tmp/syslinux -xvf /var/cache/pkg/syslinux-6.03.pkg
Now we copy the required files:

Code: [Select]

$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/core/lpxelinux.0 /usr/local/tftp/pxelinux.0
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/elflink/ldlinux/ldlinux.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/menu/vesamenu.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/lib/libcom32.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/libutil/libutil.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/com32/modules/pxechn.c32 /usr/local/tftp/
$ sudo cp /tmp/syslinux/usr/local/share/syslinux/bios/memdisk/memdisk /usr/local/tftp/
$ sudo rm -r /tmp/syslinux
At this point we have on our tftp server the files required for menus and the files required to get the clients to boot to a prompt for an installation. However we don’t have an installation media to offer yet.
For our example we are now going to get a FreeBSD 12.2 distribution iso.
Kraileth makes a great explanation of why for FreeBSD we need an mfsBSD https://mfsbsd.vx.sk/ image that can network boot.

3. Getting the image media available on our webserver.
TFTP is not meant to be used to transfer large files like distro isos. It is too slow, transferring files by limited block sizes, and the clients need to acknowledge each block, taking a long time and prone to timeouts. Fortunately we can use a local webserver. I have a XigmaNAS server that simply requires me to tick a box to enable the webserver service, choose a port and a root directory.



I chose to use http on port 8081 and the NAS has an ip of 192.168.5.5 so that’s the IP the webserver is bound to, on port 8081. We’ll need this information in a little bit.
The webroot is “/mnt/Deimos/www”  .
3.1 Now I need to connect to a shell on the NAS:

Code: [Select]

$ ssh <myuser>@nasip
Then create a subdirectory under the root:

Code: [Select]

$  sudo mkdir -p /mnt/Deimos/www/pxe
I think I had to change ownership before proceeding with subdirectories. This part is from memory, it might be needed as a later step, i.e. after the whole path was created.

Code: [Select]

$ sudo /usr/sbin/chown -R www:www /mnt/Deimos/www/pxe
Then finally creating them with a sane structure:

Code: [Select]

$ sudo mkdir -p /mnt/Deimos/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/
$ sudo fetch https://mfsbsd.vx.sk/files/iso/12/amd64/mfsbsd-12.2-RELEASE-amd64.iso -o /mnt/Deimos/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
$ sudo gzip -9 /usr/local/www/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
3.2 Now we need to edit the pxelinux.cfg/bsd file and append:

LABEL fbsd-pxe-install
   MENU LABEL Install FreeBSD 12.2 (PXE)
   MENU DEFAULT
   KERNEL memdisk
   INITRD http://192.168.5.5:8081/pxe/bsd/fbsd/amd64/12.2-RELEASE/mfsbsd.iso
   APPEND iso raw

4. Final changes in OPNSense.
This was trial and error & some additional reading online. I can’t explain how or why too well but it works for me.
I don’t use Ipv6, only v4.
I go to Services > DHCPv4 > LAN
4.1 In “TFTP server” field I entered the LAN ip of OPN: 192.168.5.1
In the “Set Bootfile” field I entered: pxelinux.0


To troubleshoot permutations of these fields I used the nmap macro that @Kraileth suggests on one is his posts to check DHCP responses. I used to get a bootfile and server offered:
From a linux machine on my lan:

Code: [Select]

$ sudo nmap --script broadcast-dhcp-discover
[sudo] password for cookiemonster:

Starting Nmap 7.60 ( https://nmap.org ) at 2021-10-01 23:13 BST
Pre-scan script results:
| broadcast-dhcp-discover:
|   Response 1 of 1:
|     IP Offered: 192.168.5.189
|     DHCP Message Type: DHCPOFFER
|     Server Identifier: 192.168.5.1
|     IP Address Lease Time: 5m00s
|     Subnet Mask: 255.255.255.0
|     Router: 192.168.5.1
|     Domain Name Server: 192.168.5.1
|     Domain Name: moomooland
|     Bootfile Name: pxelinux.0
|_    TFTP Server Name: 192.168.5.1
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 1.69 seconds
5. Testing with an actual client.
For this I used VirtualBox. I created a new Virtual machine.
Type: BSD
Version: FreeBSD (64 bit)
Memory: 1 GB; 18 GB vdi storage
Boot Order: Network, Hard Disk, Optical
Network: Bridged Adapter , Name: the name of the adapter on the host that is plugged into the LAN    that will connect to OPN.
With these settings I am saying, try to boot from the network and that network is the one where OPN is the DHCP server, giving the ip address and the network boot options.



Then I can start the Virtual Machine and verify:





At some point I had to do packet captures to verify the client to figure out that there was a tftp transfer despite the client complaining that thee was no file received.
The problem was that I had entered the “default” or “bsd” file. I didn’t realise it had to be the pxelinux.0 one that was needed in OPN.

[MrJohnBBQ]

This is gold! Thank you so much for putting it together. Exactly what I was looking for.

[cookiemonster]

Thanks for the kind feedback MrJohnBBQ. I searched high and low for instructions on this forum and other places and found none, except some hints & one user who reported he/she had done achieved success but not how.
It spurred me to do something about it. Trying to give a little back to OPN project and users, who have helped me much.

[tadchilly]

I was really excited to try this but I ran into an issue, syslinux doesn't seem to be installable via pkg anymore. I'm running    OPNsense 23.1.7_3-amd64.

Code: [Select]

root@OPNsense:/tmp # sudo pkg fetch -y syslinux
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
pkg: No packages matching 'syslinux' have been found in the repositories
Any ideas?  Thanks!

Edit:  There may be better ways to do it, but I grabbed the file from here:

Code: [Select]

wget https://pkg.freebsd.org/FreeBSD:13:amd64/latest/All/syslinux-6.03.pkg

Edit again:

AHHHHHH!!! It works!!  Over the years, I've screwed around many times trying to get a PXE server to work and never was able to until now. This was so simple!  Thank you!!

[cookiemonster]

Yes that works whilst the packages are available from the release cut of freebsd that OPN is based on, or available in repos and whether you have quarterly available, etc.
I'm glad you got it working!

[ajoeiam]

Greetings

Great stuff!!!

Except you went to BIOS boot and I've been going to UEFI boot.

Any chance you might post where changes to your script might happen in that case?

(TIA)