I've found only a few threads dedicated to this combination here (like this (https://forum.opnsense.org/index.php?topic=49067.msg248399#msg248399) and this (https://forum.opnsense.org/index.php?topic=42372.msg209571#msg209571)), and none of them give me high hopes, but maybe it's silently being worked on?
The situation is that after following the official documentation (as well as some inofficial guides (this (https://www.growse.com/2018/08/29/pxe-booting-a-raspberry-pi.html) and this (https://thainguyen.space/posts/kea-netboot/)) later on), PXE booting through KEA still only works for some clients (Realtek Boot Agent), but fails for others (NVidia PXE boot). And sadly none of the configuration parameters are self-explanatory in the GUI ("Next-server" should fill itself in automatically if TFTP boot is enabled or at least the help should mention that it's a requirement for TFTP booting), if they exist at all. Also, I was unable to get the special options from the guides to propagate to the config file, and if they did, they ended up in places different from what the inofficial guides say they should. I assumed this would be a no-brainer, as setting up Dnsmasq for PXE booting worked instantly for all clients, as did udhcpd on another machine. It's just KEA that I can't get to terms with.
So, is KEA just not ready for this, or did I simply not dig deeply enough / would have to use use the manual configuration override? I'd rather use KEA for PXE booting only, and reserve Dnsmasq for a hypothetical internal DNS service. I prefer to separate services so that I can mess up either without affecting the others. But I'd rather not mess around with half-documented hacks and an unsupported configuration in a GUI-driven environment like OPNsense. :)
I did have a user confirming the boot with the new options work as intended:
https://github.com/opnsense/core/issues/10029
Maybe try to ping them on github and see if they can help you here.
Quote from: drosophila on April 22, 2026, 03:16:28 AMPXE booting through KEA still only works for some clients (Realtek Boot Agent), but fails for others (NVidia PXE boot).
Nvidia PXE BOOT ?!
That sounds like a very old nForce4 Motherboard with their NIC and Soundcard Onboard ?!
I would simply add a nice Intel NIC to that system :)
Quote from: Monviech (Cedrik) on April 22, 2026, 07:58:44 AMMaybe try to ping them on github and see if they can help you here.
Well, I
did get the Realtek things to boot alright, just not the nVidia. However, contacting them would require me to sign up for GH, and I'd rather not have yet another account to have data stolen/sold/abused for "legitimate interest" or shady "law enforcement" demands. :)
Quote from: nero355 on April 22, 2026, 03:31:41 PMThat sounds like a very old nForce4 Motherboard with their NIC and Soundcard Onboard ?!
It's old but not
that old. :) It's an MS-7380. The NIC is an ordinary RTL8211BL but oddly they still use nVidia Boot Agent (probably because it's in their firmware anyway, as the entire chip set is nVidia). Probably that was grafted on and is the reason why it doesn't work with the standard settings.
However, every MoBo since the early 2000s has had onboard sound and NIC, except industrial and server boards of course, so that by itself doesn't seem to be an indication of age?
Quote from: drosophila on April 22, 2026, 05:47:41 PMcontacting them would require me to sign up for GH, and I'd rather not have yet another account to have data stolen/sold/abused for "legitimate interest" or shady "law enforcement" demands. :)
Proton.me account where you offload such stuff and you are done! :)
QuoteIt's old but not that old. :) It's an MS-7380.
It's horribly old : https://community.spiceworks.com/t/micro-star-international-co-ltd-ms-7380/979695
But it was/still is one of the most beautiful motherboards ever made! :)
QuoteThe NIC is an ordinary RTL8211BL but oddly they still use nVidia Boot Agent (probably because it's in their firmware anyway, as the entire chip set is nVidia). Probably that was grafted on and is the reason why it doesn't work with the standard settings.
That's one of those half this/half that constructions that use to be popular in the past.
I can't remember anymore what the exact details were, but I think your PHY part is probably made by Nvidia and the MAC part is RealTek.
Something like that anyway...
QuoteHowever, every MoBo since the early 2000s has had onboard sound and NIC, except industrial and server boards of course, so that by itself doesn't seem to be an indication of age?
I only mentioned that to check if it's a "Luxury" model or one of the cheaper models which did not have all the Nvidia features like the special Soundcard at the time and in this case also the NIC is not 100% Nvidia :)
Quote from: nero355 on April 22, 2026, 06:08:41 PMIt's horribly old : https://community.spiceworks.com/t/micro-star-international-co-ltd-ms-7380/979695
But it was/still is one of the most beautiful motherboards ever made! :)
Thanks. Now each time I boot it I hear a voice that says "it belongs in a museum". ;) But yes, it's a shame that I put it inside an opaque case, the coolers really
are lovely with all the little details. If I ever retire it, I might bolt it to a wall so it may finally shine. :)
Quote from: nero355 on April 22, 2026, 06:08:41 PMI only mentioned that to check if it's a "Luxury" model or one of the cheaper models which did not have all the Nvidia features like the special Soundcard at the time and in this case also the NIC is not 100% Nvidia :)
Like the "Karajan audio module" that went onto a special socket in one of the early 2000 boards? I wonder what connector that was, maybe something like those audio/modem risers that were popular a few years before that one?
Quote from: drosophila on April 23, 2026, 05:56:12 PMLike the "Karajan audio module" that went onto a special socket in one of the early 2000 boards? I wonder what connector that was, maybe something like those audio/modem risers that were popular a few years before that one?
My good old DFI Venus Edition had that, but what I was talking about is called
'Nvidia SoundForce' IIRC and was one of the very few that could Encode your sound so you could have Surround Sound via the Digital Output instead of the Analog Outputs in Games :)
Ah, I see, these special set-ups aren't for me so I'm rather uninformed. :) My current question about this limited edition would of course be "does that one boot from the documented KEA PXE setup?" :) Having all-solid caps in that age certainly was quite the upgrade, even without the special sound! The placement of the SB with it's little fan is kind of meh though: if the fan fails, how would you replace that special thing?
However, I just checked the datasheet of the RTL8211BL and you were right: it indeed is a plain PHY, not a fully integrated NIC, so it makes sense that the nVidia chip contains the actual MAC, therefore having the nVidia PXE boot agent is reasonable.
Quote from: drosophila on April 23, 2026, 11:33:24 PMAh, I see, these special set-ups aren't for me so I'm rather uninformed. :)
I am just geeking out a bit since that hardware era is precious to me ;)
QuoteMy current question about this limited edition would of course be "does that one boot from the documented KEA PXE setup?" :)
No idea : Sold it for big €€€ to a collector in the U.K. a couple of years ago ^_^
But I think it would because of the brand of the NICs : Broadcom or Marvell
I think those did not have any Nvidia parts involved.
QuoteHaving all-solid caps in that age certainly was quite the upgrade, even without the special sound!
I was not that special to be honest, but hey... Marketing Wizards !!! LOL!
QuoteThe placement of the SB with it's little fan is kind of meh though: if the fan fails, how would you replace that special thing?
I had cooled every part of my PC with Thermalright heatsinks at the time combined with silent GLL series Papst fans on them so no issues there ;)
See here : https://forum.opnsense.org/index.php?topic=50878.msg260535#msg260535
QuoteHowever, I just checked the datasheet of the RTL8211BL and you were right: it indeed is a plain PHY, not a fully integrated NIC, so it makes sense that the nVidia chip contains the actual MAC, therefore having the nVidia PXE boot agent is reasonable.
Thought so :)
Hehe, looks like you didn't buy the Venus to show it off, given that I can barely make it out behind all those fans and heatsinks! :D Was that a NAS with all those harddrives?
Quote from: nero355 on April 24, 2026, 12:57:09 AMNo idea : Sold it for big €€€ to a collector in the U.K. a couple of years ago ^_^
Good call, plus it's good that at least these special editions get preserved that way. Win-win, it seems! :)
Quote from: nero355 on April 24, 2026, 12:57:09 AMBut I think it would because of the brand of the NICs : Broadcom or Marvell
I think those did not have any Nvidia parts involved.
Hmm, Marvell was one but the other was a Vitesse PHY that was fed from the nForce4 and thus would likely also have used the nVidia PXE boot... I think it used the "forcedeth" driver, anyway. Oh well. :)
BTT: I believe I've found the problem: the nVidia boot agent seems to terminate the boot file name with some illegal character, or rather, with
something while there
shouldn't be anything there. The server log viewer would show just two blank lines (which is a bug in the server GUI...), but its syslog shows this:
RRQ from x.x.x.x filename /pxelinux.0<FF>The working Realtek boot agents send the same request but without the postfixed "<FF>" (which probably translates to a single byte of all ones). Obviously that should not be there but now the question is: why does it work with udhcpd and Dnsmasq, just not with KEA? Is it possible that KEA incorrectly sends this but Realtek just discards it while nVidia keeps it? Or could this be a config problem?
I seem to recall that 0xff is the UTF-8 "continuation" marker, but even if that is correct that still doesn't explain why it's there.
Quote from: drosophila on April 24, 2026, 01:49:46 AMHehe, looks like you didn't buy the Venus to show it off, given that I can barely make it out behind all those fans and heatsinks! :D
Was that a NAS with all those harddrives?
I just had a lot of storage for that time and it was mostly used for gaming :)
2 x WD Raptor 36 GB ADFD Nvidia RAID 0 for Windows
2 x WD Raptor 150 GB ADFD Nvidia RAID 0 for Games
2 x 250 GB Hitachi HDD for Storage
2 x 250 GB Samsung HDD for Backup of the Storage
I have always been a fan of a lot of storage so after thuis build it only got worse up to a beautiful AsRock X79 Extreme11 with 14 SATA Ports :P
QuoteHmm, Marvell was one but the other was a Vitesse PHY that was fed from the nForce4 and thus would likely also have used the nVidia PXE boot... I think it used the "forcedeth" driver, anyway. Oh well. :)
IIRC driver-wise there was not much difference between the two NICs in Windows at least...
Not sure about Linux/*BSD because even when using some kind of Live CD or whatever option there was at the time I have never done anything with PXE Boot or paid attention to the
dmesg output for example.
QuoteBTT: I believe I've found the problem: the nVidia boot agent seems to terminate the boot file name with some illegal character, or rather, with something while there shouldn't be anything there. The server log viewer would show just two blank lines (which is a bug in the server GUI...), but its syslog shows this:
RRQ from x.x.x.x filename /pxelinux.0<FF>The working Realtek boot agents send the same request but without the postfixed "<FF>" (which probably translates to a single byte of all ones). Obviously that should not be there but now the question is: why does it work with udhcpd and Dnsmasq, just not with KEA? Is it possible that KEA incorrectly sends this but Realtek just discards it while nVidia keeps it? Or could this be a config problem?
I seem to recall that 0xff is the UTF-8 "continuation" marker, but even if that is correct that still doesn't explain why it's there.
Would it be an option to have a dedicated PXE Boot VLAN on your network ?
In the past I have worked for a company that had this and the software doing the PXE Boot stuff was some kind of dedicated Linux distro at the time.
So in this case OPNsense would be just the Router providing internet access for all those NetBoot images :)
If a potential bug crystalizes out here, you can always ask on the kea mailing list or their gitlab.
Just recently another user found something else and we guided them to kea and their issue has been acked it seems.
Some reference:
https://github.com/opnsense/ports/issues/267
After looking at some packet dumps it seems like there are at least two methods of sending the boot file name. Obviously, KEA and udhcpd use different ones, apart from ordering packet options differently.
KEA sends: ... 43
0b 2f 70 78 65 6c 69 6e 75 78 2e 30
ff 63 ...
which translates to "/pxelinux.0" but as you can see, there is an "ff" at the end. This is not a null-terminated string, instead the string length is prefixed, "0b" (decimal 11) in this case.
The decoded packet reads:
e.e.e.e.67 > 255.255.255.255.68: xid:<censored> flags:0x8000 Y:c.c.c.c S:b.b.b.b ether <censored> vend-rfc1048
DHCP:OFFER SM:255.240.0.0 DG:g.g.g.g NS:e.e.e.e DN:"local" LT:3116 SID:e.e.e.e BF:"/pxelinux.0" (DF) [tos 0x10]
udhcpd sends: ... 00
00 2f 70 78 65 6c 69 6e 75 78 2e 30
00 00 ...
which, again, translates to "/pxelinux.0" but is surrounded by all zeros so it is, intentionally or not, a null-terminated string.
The decoded packet reads:
e.e.e.e.67 > 255.255.255.255.68: xid:<censored> flags:0x8000 Y:c.c.c.c S:b.b.b.b ether <censored> sname "<censored>" file "/pxelinux.0" vend-rfc1048
DHCP:OFFER SID:e.e.e.e LT:864000 SM:255.240.0.0 DG:g.g.g.g NS:e.e.e.e DN:"local"
So while KEA passes the filename in a "BF" parameter, udhcpd supplies it in a "file" parameter.
So, given the correct string length passed by KEA it seems like the bug is in the nVidia boot agent, which, of course, is the worst possible outcome because that means that these will never work with KEA.
I
might find a way to make KEA send a "file" parameter instead of the "BF", hoping that it will also surround it by zeros or that it will just be implemented properly. More likely I'll ditch KEA for Dnsmasq because naturally I want a solution that works in all cases. The only upside of this is that now I know it.
Quote from: nero355 on April 24, 2026, 03:39:20 PMWould it be an option to have a dedicated PXE Boot VLAN on your network ?
In the past I have worked for a company that had this and the software doing the PXE Boot stuff was some kind of dedicated Linux distro at the time.
So in this case OPNsense would be just the Router providing internet access for all those NetBoot images :)
That would be an excellent option for actual thin clients, as this kind of boot is quite insecure and should ideally be confined even on the LAN.
However, I use PXE for all sorts of things
except thin clients. :) (my infrastructure is not robust enough; to depend on it, I'd need HA first) Like initial / post-modification memtest86+ tests and serving the OS installers (debian netinstall), and that can potentially be every machine (except the TFTP server itself, of course). That way, I don't have to fiddle with USB sticks and can apply the same procedure to other peoples machines if need be. :) That's why I need it to "always work", as the type of client is not forseeable. One thing I still need to figure out is how to chainload the BSD bootloader from pxelinux (which provides the menu). Is there some sort of "GRUB" for PXE?
Quote from: drosophila on April 25, 2026, 02:48:01 AMHowever, I use PXE for all sorts of things except thin clients. :)
So did they! :)
In fact mainly this :
QuoteLike initial / post-modification memtest86+ tests and serving the OS installers (debian netinstall)
But I can remember that at some point it was no longer compatible with UEFI Servers so double check if the software you eventually go for can also do that!
QuoteOne thing I still need to figure out is how to chainload the BSD bootloader from pxelinux (which provides the menu).
Is there some sort of "GRUB" for PXE?
I can remember they use to do that a bit different than all the Linux distros but I don't know the exact details anymore...
There was "something else" that "kickstarted" the whole thing I believe...
I haven't kept up with netboot for freeBSD but in the past it needed to use mfsBSD. Check https://forum.opnsense.org/index.php?topic=25003.msg120021#msg120021 for some pointers.
FreeBSD can perfectly well netboot if you serve the root FS via NFS as intended and documented. No mfsBSD, no pxelinux, ... just an NFS based root filesystem like BSD and Sun have been doing for decades.
Blog post in German:
https://punkt.de/de/blog/2017/automatisierte-installation-von-servern-mit-freebsd-und-zfs.html
Quote from: nero355 on April 25, 2026, 03:15:13 PMBut I can remember that at some point it was no longer compatible with UEFI Servers so double check if the software you eventually go for can also do that!
That would likely be a reason to keep trying to get KEA to work; possibly I'll attempt to install a symlink to the actual install file that with an appended "ff". Should be possible but it's certainly an evil hack. KEA can easily filter by boot agent type and therefore serve different stuff for BIOS vs. EFI.
Quote from: https://punkt.de/de/blog/2017/automatisierte-installation-von-servern-mit-freebsd-und-zfs.htmlDie bereits per TFTP bereitgestellte Umgebung muss nun noch zusätzlich per NFS exportiert werden.
That likely was the issue that kept my attempt from succeeding. It did load the netboot-capable boot but since I expected this to be fully handled by TFTP I didn't export the boot FS via NFS (which would have been easy since NFS already serves selected datasets). A pity that one can't do away with TFTP entirely, NFS is less hacky and much faster. and if you have TFTP running, then things must already be secured against nefarious clients so that NFS won't be an additional security issue. For installers, one can just set all respective underlying filesystem permissions to be readonly for everyone since they're not sensitive and thus only manipulation must be prevented. The automated install might need sensitive things like passwords or certificates to be served ("Anschließend setzen wir das Root–Passwort, damit wir uns über die Konsole anmelden können: (...) ROOT_PW_HASH ist dabei natürlich durch den echten Hash des Root–Passworts zu ersetzen."), but a basic interactive install doesn't have these problems.
Thanks for the heads-up!
Just as (final?) info: the evil hack works: I created a symlink with a final character of 0xff, that points to another symlink with the same name except for the appended 0xff, which in turn points to the intended bootfile (so I can simply change this intermediate symlink, and PXE loaders without this bug will use that, so all end up loading the same file). Then I set KEA to serve the filename without the appended 0xff. The nVidia PXE boot agent boots just fine now, so that really is the entire problem.
Now the only thing I could do is to try to find out whether the 0xff that KEA sends after the end of the BF parameter is somehow necessary or not, and if not, try to convince the KEA developers to change that into an 0x0, hoping that the nVidia PXE boot agent would then view this as null-terminated string and handle it properly. Or just keep the evil hack, which of course stinks in terms of maintainability. :(
Quote from: Patrick M. Hausen on April 25, 2026, 10:05:10 PMFreeBSD can perfectly well netboot if you serve the root FS via NFS as intended and documented. No mfsBSD, no pxelinux, ... just an NFS based root filesystem like BSD and Sun have been doing for decades.
Blog post in German:
https://punkt.de/de/blog/2017/automatisierte-installation-von-servern-mit-freebsd-und-zfs.html
Very nicely written resource, thank you Patrick. I have bookmarked for my next time I need to re-do it. I've been meaning to find the time to investigate doing the efi pxie boot. Much obliged.
Alright, this has not been the final word. Instead I tested some more with appending a true zero. Doing so through the GUI seems to be impossible because it always escapes the escape character "\" into "\\" and therefore destroys the escaping, but from the config file itself it works with an appended "\u000". But for some reason this works only if at least one letter follows the true zero, otherwise KEA just ends there and doesn't send the zero at all. Adding any character(or multiple) makes it send the zero. Like this:
43 0e 2f 70 78 65 62 6f 6f 74 66 69 6c 65 00 56 ff 23 94
That translates to "/pxebootfile<zero>V", note how the length field now is 0e, correctly counting the length including the zero and the appended "V". This is handled correctly by both Realtek and nVidia, both end up requesting "/pxebootfile", which is correct. So is KEA wrong in not sending the final zero even though it correctly specifies the length? Did nVidia work off an outdated spec or did they simply commit the basic blunder of counting from zero instead of one? Does Realtek have a more stringent sanitizer that converts all non-ASCII characters into end-of-string?
Sadly, the "specification" in RFC4578 only says "The format and contents of these options are NOT defined by the PXE specification.", so this must be defined outside of an RFC.
IBM has a document here (https://dimitrije.website/files/pxespec.pdf) that lists "UINT8 bootfile[128]; Bootfile: Boot file name. Null terminated string." on page 50. So it seems like the error is indeed on KEAs part, but what about the difference in parameters (noted above: "file" vs. "BF")? Is there yet another spec that supersedes the 1999 IBM document?
So it seems there is one from uefi here (https://uefi.org/specs/UEFI/2.10/24_Network_Protocols_SNP_PXE_BIS.html). They don't want to let me look at their precious document through my secured browser, and I'm not willing to compromise my anonymity, especially not to satisfy some random sites self-importance. So my research ends here for now. Maybe I'll get back to it on someone elses computer, who doesn't care about security and privacy.
The final question is: does the GUI have some means to escape escape characters so that it does not escape escape characters? IOW: how do I get the "\" into the config file through the GUI without it being mangled into "\\"? So that the resulting config file reads: "data": "\/pxebootfile\u0000V"Not relying on evil filesystem hacks is nice, but managing KEA through files while a perfectly capable GUI is there for everything else also detracts from maintainability.
Since you are already way into hex anyway, why not send hex directly?
Its possible with the new options menu, just select hex.
Yup, that works, thanks for the heads-up! :) That way I can filter by EFI / BIOS like it's supposed to be done though the GUI, only the filenames are less readable but much better than manual file alterations with unconventional characters. This way, at least it's clear what's going on. I'll settle for this. :)
Quote from: Monviech (Cedrik) on May 03, 2026, 07:12:35 AMSince you are already way into hex anyway, why not send hex directly?
Because I had assumed that would only take a single byte or somesuch. :)
The format could be explained a little better in the help resp. error text, to make clear it expects something like "2f707865626f6f7466696c650056" (which ends up verbatim in the config file). ;)
Quotepayload in hexadecimal byte pairs.
The error text
QuoteHex value must contain valid hexadecimal byte pairs.
To input a series of un-delimited and un-prefixed hex characters was unexpected, given that most (all?) other tools delimit them by a space save for MAC addresses where they use a colon. Maybe the text could read "payload in hexadecimal byte-wise representation, without delimiters and prefixes."?
Would be neat to have a "comment" field per option where I could put the reminder "For nVidia to work, must always append "0056" to hex value!" or somesuch. ATM, I made an option that has this in its description to serve this purpose. :)