GUI/Shell crashing

[Mattps]

Hi Forum,

I need a some troubleshooting an unstable OPNSense installation. I've moved from a virtualised OPNSense instances to running of dedicated hardware:

HP T730 Mini PC 8GB RAM, 64GB NVMe, Intel Pro/1000 ET 82576 Quad NIC

I'm using the latest VGA image with no additional plugins. OPNSense will suddenly lock up, GUI/Shell become unresponsive and the NIC ports link lights stop. Rebooting the PC resolves the issue. Yesterday with just 1 LAN and WAN interface configured it would crash after about 20 mins. I installed Windows 11 IoT over the top, updated the BIOS, ran hardware diagnostics - all passed. With Windows it ran without issue. Confident that it was fixed I reinstalled OPNSense and it seemed to work - ran for over 90 mins without issue.

Today I have configured 3 additional LAN interfaces and the problem is back, except now it will stay on for about 2 mins before dying. Any had any experience of this or where I can check logs to see what is going on?

Thanks in advance,
Matt

[meyergru]

Try using the tuneable "hw.pci.enable_aspm = 0" to disable ASPM if your BIOS does not support it. Those freezing issues often point to ASPM issues.

[Mattps]

Thanks Meyergru, I'll try that. I managed to get the Firmware > Report output before it crashed, I've attached in case that's helpful.

[Mattps]

It stayed up for about 20 mins and then died again unfortunately.

[meyergru]

Did you try installing the microcode updates? It does not look like it from the report...

There is definitely something off in the Power Management in your firmware:

[1] Firmware Warning (ACPI): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20221020/tbfadt-796)

If the uptime is exactly 20 Minutes, I would look for a BIOS watchdog. When you look at the forum search for HP T730, you will find a few other reports of systems freezing or crashing.

[Mattps]

I believe the microcode update is in the latest BIOS update, which I have installed. I'll have a look at the other posts. Do you think this is an issue with the HP T730 or the PCIe NIC?

[meyergru]

I really do not know.

[Mattps]

Well, no closer to getting this working. I reimaged the T730 with Windows 11 IoT today and ran non-stop speed tests for 6 hours and it didn't skip a beat. Then I re-imaged with opnsnese and just left the LAN connection in ping a host. This worked for 2.5 hours and stayed up. I then connected the LAN and tried to run a bandwdidth speed test and bang it locks up. I thought it may have been the onboard Realtek nic so changed interfaces to just use the Intel pro/1000 but tis made no difference.
I can't even do an opnsense fw update before it dies, it gets about 10 secs in the locks up.

Could this be a compatibility issues with FreeBSD and an HP T730 think client? I'm sure I read other people using this device with opnsense.

Not sure where to go from here.

[meyergru]

RealTek NICs are known to work badly with FreeBSD / OpnSense. If at all, you can try the os-realtek-re plugin.

I also do not know if the latest BIOS is up to par w/r to microcode updates (or if there are still updates from AMD for this old platform).

And, yes of course it can be a compatibility issue. FreeBSD does not support as many hardware types as Linux and some of the FreeBSD drivers are abysmal.

[Mattps]

Microcode updates are applied via a BIOS update, there aren't any separate updates. It's running the lasted BIOS L43 1.16.

I would try the Realtek plugin but this requires to upgrade the opnsense fw first and I can't get the device to stay up long enough.

It's just strange that it seems to die under load.

[meyergru]

Microcode updates are applied via a BIOS update, there aren't any separate updates. It's running the lasted BIOS L43 1.16.

Some things to clear up here:

1. I am not saying that there is a newer microcode update - what I do say is that IMHO, manufacturers are slow to adapt the newest microcode updates.

2. The BIOS you are using is at least 3 years old: https://h30434.www3.hp.com/t5/Desktop-Operating-Systems-and-Recovery/HP-T730-Bios-update-failed/td-p/8453495

3. Yes, the microcode updates delivered as OS packages are separate updates, which can be significantly newer than those delivered in your BIOS. And they are needed for some platforms, like the N1x0 and other 12th gen Intel chips with OpnSense from 25.7. upwards, see: https://forum.opnsense.org/index.php?topic=42985.0, point 23.

That being said, IDK if there actually are any updates available or if they change anything for your symptoms. I just would not shrug this off if I were you.

[Mattps]

Thanks Meyergru,

I've looked and couldn't find any microcode updates AMD only deliver these for this CPU via bios updates and the bios update for this model is only delivered by HP.

I'm going to put Proxmox on it today and try and run OPNsense as a VM and see if it's stable. The goal here to was to get it running natively on separate hardware from my larger Proxmox servers, so although this would be a compromise, I can live with for now.

[Patrick M. Hausen]

I've looked and couldn't find any microcode updates AMD only deliver these for this CPU via bios updates

Install the os-cpu-microcode-amd plugin and reboot to receive the latest microcode updates.

[meyergru]

I've looked and couldn't find any microcode updates AMD only deliver these for this CPU via bios updates and the bios update for this model is only delivered by HP.

That is only partially correct. AMD may deliver what they want. The updates contained in BIOSes are being extracted and put into separate packages, such as os-cpu-microcode-amd for OpnSense, to be applied apart from BIOS updates. BTW: There are similar packages for Linux / Proxmox as well using the same extracted firmwares.

I repeatedly tried to tell you. Had you looked at https://forum.opnsense.org/index.php?topic=42985.0, point 23 and followed the link to the official docs there, you should have noticed.

The only question is if there is actually an update available in that package for you specific CPU and if it fixes your problem. You will find out only if you try, not by discussing if this is possible at all, so please do as Patrick said.

[Mattps]

Thank you,

I did try the tuneable (point 23) as you suggested previous - as indicated in post #3. Sorry, only just spotted Patrick's reply - this was installed before creating this post.