Messages

That sounds a little like MAC spoofing doesn't work properly on the default Realtek drivers - sending may work but the NIC might just drop received packets as "not my MAC". The vendor driver may do it correctly. Why this works without spoofing now suggests the reports are correct but why it would not do anything with the default driver is odd in that case. Maybe some leftover from the previous config. Backing up the working config will safeguard you from any config missteps you might do. Also, you could attempt to use a fresh -nano on an USB stick to see if a clean OPNsense install would do anything differently to rule out configuration cruft.

in-line Ethernet surge protection
(...)
The 2.5GbE port on the modem was problematic

That makes sense: the surge protection adds varistors or / and discharge tubes / or / and "high voltage zeners" towards ground or / and across pairs. This adds capacitance and possibly non-linear leakage currents, both of which degrade signal integrity. These ports probably aren't made with 2.5GBps in mind and depending on age may be lucky if they were meant for 1GBps. The internal routing may also not even meet Cat6 requirements.

My current cable modem (an ISP-issued replacement, newer model) doesn't seem to care and the 2.5GbE port negotiates fine.

Your new modem may have better PHYs or just care less about such issue. Or the old one might just have had degraded components (age) that already led to degraded signal integrity and the additional effects of the surge protection just tipped it out of tolerance.

If that is a grounding issue, then it would be possible to confirm this by connecting only the outer shell of the ISP coax to the modem, leaving the inner connection out. For example by bringing the plug and socket together on the outside only, fixating by rubber bands or something else that creates a bit of spring tension. This would eliminate the possibility of your ISP using TR-069 to mess up your devices, as it would leave the entire broadband link down, retaining the exact same logical case as with the fully unplugged coaxial cable.

However, both your Protectcli box and the Asuswrt have an external PSU that doesn't connect the ground pin, at least that's what I get from the ad pictures. Their respective external PSUs may still differ electrically but these devices themselves don't seem to be grounded.

[hw.re.s5wol]

There are all sorts of reasons to why you'd want to do that, be it on the WAN or the LAN. Waking up other things is easily configured through the GUI, but WOL the sensebox itself (or disabling that) doesn't seem to be.

Naturally, you need to first enable "Wake on PCIE cards" in the MoBo firmware, possibly even disable "Deep S5", unless it has an "auto" setting that disables it if the PCIe wake is enabled. Some MoBos hide that behind a setting called "ERP" or "EUP" and provide no "auto" setting, so "disable" is the only way to make it work. Naturally, this will consume more power in the "off" state, but there is no better way to do it. You should disable all other wake-up events that you're not using, but whether or not that actually saves any power will depend entirely on your hard- and firmware.

Anyway: ifconfig will probably return the seemingly proper "options=2008<VLAN_MTU,WOL_MAGIC>" out of the box, but what is missing is to set two tunables, namely hw.re.s5wol and hw.re.s0_magic_packet, to actually make it work. Nothing will change in the ifconfig output after setting these and rebooting, but WOL should now work as expected.

Do note that this means that WOL is enabled on all interfaces, including WAN, so anyone who manages to send the magic packet to one of these will be able to power up your box. However, remotely powering up your firewall isn't going to offer any gain unless the attacker is your local electricity provider. There probably are options for the secure WOL if your NIC supports it, but magic packet works with any WOL program, including OpenBSDs "arp -W". If you're worried about IoT devices (even if not hacked yet) or your offspring abusing this to get internet access, you need stricter firewall rules. :)

Hm. Vielleicht ist das ja genau das Problem, daß VLAN7 in diesem Fall nicht stimmt? Kann man das in der Fritze irgendwo nachsehen, oder muß man das sniffen?

1und1 resellt alles, was irgendwo rumliegt. :) Hier wurde z.B. von Deutsche Glasfaser verlegt und 1und1 vermarktet das. AFAICS hat die DG das exakt selbe oder zumindest extrem ähnliche System wie Lünecom, was hier diskutiert wurde. Ich habe zumindest beim Reinschauen dasselbe gesehen, inklusive der Freischaltung über einen Server im 10./8 Netz usw. . Also könnte es schon sowas sein, allerdings müßte dann auch die Fritze erst freigeschaltet werden gemußt haben.

Good detective work! This seems to indicate that there is some cruft stored in the config files, possibly leftovers from old configuration changes, now masked by their wrapper services being disabled. IDK why sense would store ARP things in the config, but it might be old entries for IP assignments on some obscure sub-service page, or maybe even MAC-based rules for the firewall that have a side-effect. Fixed ARP entries are possible at least in the config files (you can manually add ARP entries from the console, so scripts can also do it). The fact that it comes back only after some time hints that it might coincide with provider-induced WAN IP reassignments, resulting in the respective services are being refreshed.
Of course, it is possible to assign arbitrary MACs to interfaces in most OS/drivers, but that would also have been set up manually, just like custom scripts.

It might work to "clean up" the config by exporting it, clearing the current one, then re-importing the previously exported one. That would get rid of everything that is in auto-generated config files but not in the actual config. You could however grep through the generated xml and see if those MACs appear in it, and if so, where.

all devices both wired and wireless can't seem to do anything during this

First thing to check: what do the lights do? Do switch / NIC indicate heavy traffic? No traffic at all? Normal traffic?
Second thing to check: can they communicate among themselves? IOW, on their segment / switch?

, I am leaning toward it being a WAN-related issue as on the odd occasion when the freeze up has been long enough, I am still able to log into the Admin portal on my router.

So the LAN part is perfectly fine (unless you're using a dedicated admin port?), and the devices on your LAN can probably communicate normally among themselves. And the Protectli isn't frozen either, nor is the network stack. Keep the dashboard open and observe it for CPU / RAM / whatever spikes that shouldn't be there when the lockup happens. You probably need to add a couple of useful widgets first, like "CPU", "Traffic Graphs" and "Thermal Sensors".

I have determined the issue is my OpnSense router as I have removed it from my network and then ran each of the ISP modem and Amplify-HD as the router for a day each and during those two days I did not have any of the freezes. I have also taken the extreme move of completely wiping my router and just having it run as it comes out of the box, just as a DHCP server, no ZenArmour or OpenVPN etc. and I still get the freezing. No matter what configuration I run my network in, as soon as OpnSense is the router, the freezing happens.

That rules out updates for ZA or other blocklists clogging up the machine. I'd look for WAN-related events like IP address changes, possibly interface-related if we assume that the WAN interface might simply have a defect. Would it be possible to reassign the WAN and one of the other interfaces (the box has 4 AFAICS) to see if the issue persists unchanged (so, on WAN) or sticks to the interface?

Also, you could observe the "Live view" on the Firewall. You don't need to interpret the individual lines, just look for changes in pattern (note that to do so, you'd need to stare at those logs while everything is normal for a while to be able to see what the normal patterns might be: not every "wall of red lines" indicates something unusual.). You might need to first enable logging for all rules first. Familiarize yourself with the settings and buttons on that page so you can hit the "stop" button in time, possibly increase the "Table size" to 100 or even more for that.

Since that is an Intel, you should install the "os-cpu-microcode-intel" plugin, even though this doesn't seem to be CPU related.

I would set up a machine that is normally on anyway (like your desktop / work machine) to do a continuous ping (not floodping, just a friendly once-per-second endless ping (/t in Windows, I believe)) to one of your other "always on" LAN devices (printer, TV, smartbulb, home automation, toaster, ...) in one terminal, and another such ping to something on the internet that won't be going anywhere, say, www.microsoft.com. Possibly a third terminal pinging the LAN interface of your Sensebox for good measure.

Just keep them running until the event and see which one starts failing / changes behavior, and how, once it occurs. Or even if at all: if it is a DNS issue, running pings won't be affected.

Hab das gestern auch mitbekommen, glücklicherweise hatte ich an der Sensebox seit Tagen nichts gemacht und darum das Problem sofort für eine externe Störung gehalten und nur ganz allgemein hier reingeschaut. Dann tauchte auch schon hier Dein Thread auf und ich hatte was zu Lesen. :)

Yup, that works, thanks for the heads-up! :) That way I can filter by EFI / BIOS like it's supposed to be done though the GUI, only the filenames are less readable but much better than manual file alterations with unconventional characters. This way, at least it's clear what's going on. I'll settle for this. :)

Since you are already way into hex anyway, why not send hex directly?

Because I had assumed that would only take a single byte or somesuch. :)
The format could be explained a little better in the help resp. error text, to make clear it expects something like "2f707865626f6f7466696c650056" (which ends up verbatim in the config file). ;)

payload in hexadecimal byte pairs.

The error text

Hex value must contain valid hexadecimal byte pairs.

To input a series of un-delimited and un-prefixed hex characters was unexpected, given that most (all?) other tools delimit them by a space save for MAC addresses where they use a colon. Maybe the text could read "payload in hexadecimal byte-wise representation, without delimiters and prefixes."?

Would be neat to have a "comment" field per option where I could put the reminder "For nVidia to work, must always append "0056" to hex value!" or somesuch. ATM, I made an option that has this in its description to serve this purpose. :)

Das kommt mir irgendwie vor als könnte es sich nicht entscheiden, welches Interface das Richtige ist. Ist das eine komplett neue Konfiguration, oder wurden da evtl. mal die Interface-Assignments getauscht, so dass nun in einer aktuell nicht sichtbaren Konfigurationsstelle das falsche Interface steht, so dass es eine Race-Condition gibt? Kann man z.B. mal die beiden Interface-Assignments zwischen LAN und WAN tauschen (und natürlich die Kabel umstöpseln), um zu sehen, ob das Problem dann auch auftritt?

[But]

Alright, this has not been the final word. Instead I tested some more with appending a true zero. Doing so through the GUI seems to be impossible because it always escapes the escape character "\" into "\\" and therefore destroys the escaping, but from the config file itself it works with an appended "\u000". But for some reason this works only if at least one letter follows the true zero, otherwise KEA just ends there and doesn't send the zero at all. Adding any character(or multiple) makes it send the zero. Like this:

43 0e 2f 70 78 65 62 6f 6f 74 66 69 6c 65 00 56 ff 23 94
That translates to "/pxebootfile<zero>V", note how the length field now is 0e, correctly counting the length including the zero and the appended "V". This is handled correctly by both Realtek and nVidia, both end up requesting "/pxebootfile", which is correct. So is KEA wrong in not sending the final zero even though it correctly specifies the length? Did nVidia work off an outdated spec or did they simply commit the basic blunder of counting from zero instead of one? Does Realtek have a more stringent sanitizer that converts all non-ASCII characters into end-of-string?

Sadly, the "specification" in RFC4578 only says "The format and contents of these options are NOT defined by the PXE specification.", so this must be defined outside of an RFC.
IBM has a document here that lists "UINT8 bootfile[128];   Bootfile: Boot file name. Null terminated string." on page 50. So it seems like the error is indeed on KEAs part, but what about the difference in parameters (noted above: "file" vs. "BF")? Is there yet another spec that supersedes the 1999 IBM document?
So it seems there is one from uefi here. They don't want to let me look at their precious document through my secured browser, and I'm not willing to compromise my anonymity, especially not to satisfy some random sites self-importance. So my research ends here for now. Maybe I'll get back to it on someone elses computer, who doesn't care about security and privacy.

The final question is: does the GUI have some means to escape escape characters so that it does not escape escape characters? IOW: how do I get the "\" into the config file through the GUI without it being mangled into "\\"? So that the resulting config file reads:

Code Select Expand

"data": "\/pxebootfile\u0000V"Not relying on evil filesystem hacks is nice, but managing KEA through files while a perfectly capable GUI is there for everything else also detracts from maintainability.

Just as (final?) info: the evil hack works: I created a symlink with a final character of 0xff, that points to another symlink with the same name except for the appended 0xff, which in turn points to the intended bootfile (so I can simply change this intermediate symlink, and PXE loaders without this bug will use that, so all end up loading the same file). Then I set KEA to serve the filename without the appended 0xff. The nVidia PXE boot agent boots just fine now, so that really is the entire problem.

Now the only thing I could do is to try to find out whether the 0xff that KEA sends after the end of the BF parameter is somehow necessary or not, and if not, try to convince the KEA developers to change that into an 0x0, hoping that the nVidia PXE boot agent would then view this as null-terminated string and handle it properly. Or just keep the evil hack, which of course stinks in terms of maintainability. :(

But I can remember that at some point it was no longer compatible with UEFI Servers so double check if the software you eventually go for can also do that!

That would likely be a reason to keep trying to get KEA to work; possibly I'll attempt to install a symlink to the actual install file that with an appended "ff". Should be possible but it's certainly an evil hack. KEA can easily filter by boot agent type and therefore serve different stuff for BIOS vs. EFI.

Die bereits per TFTP bereitgestellte Umgebung muss nun noch zusätzlich per NFS exportiert werden.

That likely was the issue that kept my attempt from succeeding. It did load the netboot-capable boot but since I expected this to be fully handled by TFTP I didn't export the boot FS via NFS (which would have been easy since NFS already serves selected datasets). A pity that one can't do away with TFTP entirely, NFS is less hacky and much faster. and if you have TFTP running, then things must already be secured against nefarious clients so that NFS won't be an additional security issue. For installers, one can just set all respective underlying filesystem permissions to be readonly for everyone since they're not sensitive and thus only manipulation must be prevented. The automated install might need sensitive things like passwords or certificates to be served ("Anschließend setzen wir das Root–Passwort, damit wir uns über die Konsole anmelden können: (...) ROOT_PW_HASH ist dabei natürlich durch den echten Hash des Root–Passworts zu ersetzen."), but a basic interactive install doesn't have these problems.

Thanks for the heads-up!

I see, either way makes sense. It seems like I could implant the CPU type in the PLUGIN_VARIANT variable by deducing it from dmesg output, but that would probably be brittle, even if it can run early enough.