Lenovo M720q randomly hangs during headless cold boot/reboot

[userunkn4wn]

Hi,

I'm having a strange boot issue with OPNsense on a Lenovo ThinkCentre M720q Tiny and I'm trying to determine whether this is related to FreeBSD/OPNsense, EFI console handling, or the Lenovo firmware.

Hardware / system:

- Lenovo ThinkCentre M720q Tiny
- Intel Core i5-8400T
- 8 GB RAM
- Toshiba NVMe SSD
- Intel I350-T4 PCIe NIC
- Onboard Intel I219-V
- Intel Dual Band Wireless-AC 8265 (currently unused)
- OPNsense 26.7.3_8
- UEFI boot
- Latest Lenovo UEFI/BIOS installed (the same problem was already present with the previous UEFI/BIOS version, so updating to the latest firmware did not resolve it)

The problem is that the machine sometimes hangs during boot or reboot when running headless.

When a monitor is connected via DisplayPort before powering on/rebooting, the machine appears to boot/reboot reliably. Without a monitor, cold boots and reboots can randomly hang.

When it hangs, OPNsense does not become reachable over the network. Connecting a monitor after the machine has already started/hung does not produce any video output either — the monitor has to be connected before boot for video output to work.

Interestingly, after a failed boot, forcing the machine off and starting it again will often result in a successful headless boot.

After some research, I came across the following article describing a very similar issue with OPNsense systems not booting without a monitor:

https://www.thomas-krenn.com/en/wiki/OPNsense_does_not_boot_without_monitor

Following the recommendation in that article, I changed the console configuration to:

    Primary Console: Serial Console
    Secondary Console: VGA Console

Subjectively, this seemed to make the system somewhat more reliable, although the boot issue was still present. The reboot could still hang; however, in this configuration a short press of the power button was sufficient to shut the machine down from the hung state, whereas previously I sometimes had to hold the button for a forced power-off.

During boots/reboots with a monitor connected, however, I noticed the following messages from the FreeBSD loader:

    no valid consoles!
    console vidconsole is unavailable

The loader also listed the available consoles as:

    Available consoles:
        efi
        eficom
        comconsole
        nullconsole
        spinconsole

Since "efi" was available while "vidconsole" was explicitly reported as unavailable, I changed the configuration again to:

    Primary Console: EFI Console
    Secondary Console: None

With this configuration, the console warnings disappeared completely, and headless reboots also seem to have become more reliable. However, the underlying problem is not completely solved: cold boots without a monitor can still occasionally hang, while other cold boots with exactly the same configuration succeed normally.

Using EFI Console therefore appears to be the more appropriate console configuration for this UEFI system, but I am not yet sure whether the console configuration is actually related to the root cause of the intermittent headless boot issue.

The particularly strange behavior is:

    Monitor connected before boot:
        boot/reboot appears reliable

    No monitor:
        cold boot/reboot may hang
        OR may boot completely normally

    Failed headless boot:
        power off -> another headless cold boot often works

The system is already running the latest Lenovo UEFI/BIOS. The issue also occurred with the previous firmware version, so this does not appear to be a regression introduced by the latest UEFI update.

At this point I have left the console configuration at:

    Primary: EFI Console
    Secondary: None

since this matches the consoles actually provided by the UEFI loader and avoids the "vidconsole is unavailable" warning.

Has anyone seen similar behavior with the Lenovo M720q/M920q or other Intel Coffee Lake systems running FreeBSD/OPNsense?

Any suggestions for further diagnostics would be appreciated. I would prefer to identify where the boot actually stops before randomly changing hardware or loader tunables.

Thanks!

[MoonbeamFrame]

Not quite your problem, but I've had some Linux servers/clients where not having a monitor connected caused issues.

I resolved them but using a display emulator plugged into a video port.

e.g. DisplayPort display emulator/EDID emulator Plug

[userunkn4wn]

Yes, a display emulator would be a relatively easy but not a really satisfying fix, but yeah, I will consider this when I run out of options.

[Colt45]

Ive had this same issue on and off with one of my machines. A Dell Optiplex 3040 with an i5-6500 in my case.
Unfortunately Ive not been able to discover what the issue is other than plugging some sort of display does cause it to boot. Looking at what is on the screen when it does 'hang' it is past EFI and the bootloader, so I think it is some sort of kernel issue.
Maybe something with this machine is causing it to believe there is a display connected and its stuck trying to read EDID or something. Im not sure. I wouldnt think that this would stop the boot up process but it is.

I made the mistake of upgrading my apartment router remote one day and it never came back up after, which is how I discovered the issue. It wasnt until the second upgrade that I put two and two together. I plugged in HDMI to see what was up, rebooted and it came up immediately. I was then able to replicate the hang by trying to boot with HDMI disconnected.

[Seimus]

This sounds like the BUG with Intel iGPUs on Linux.
I have had this with one of my Proxmox nodes, where the issue was not during the reboot cycles but during usage of the iGPU and a deep power state.

Is it possible if not already that you try to disable the iGPU(but if you do that you may not have any video output) in BIOS or at least look at the C states and disable them for a test.

Note here that in order for intel to use Turbo, you need to have at least a C3 state.

Regards,
S.

[MeltdownSpectre]

Hardware wise my M720q is pretty close to yours.

If I have a monitor connected before boot, it boots up fine with a working output on the monitor. If I boot without a monitor and plug in the HDMI cable while it's already running, there is no display output.

However, I do not have any cold boot issues to report. Mine works perfectly fine headless and has been for almost 3 years now. I'm not on the latest BIOS, I believe 1 or 2 versions behind the latest, ending in '77A'. I think it was released some time in 2024.

[Patrick M. Hausen]

Some motherboard/BIOS combination present no EFI console at all when there is only an e.g. HDMI connector, no serial port, no IPMI ... and no monitor plugged in.

The FreeBSD boot loader crashes when trying to boot with EFI console and the list of console devices provided by the BIOS is empty.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288900

The known workaround is to set primary console to serial ("comconsole") and secondary console to none. See the linked bug report. The downside is that you won't have console output when you do connect a monitor, unless you interrupt the boot loader early on and undo the setting.

HTH,
Patrick

[talespin]

Hi,

I am using M720q as a router with OPNSense with no monitor connected.

M720q
8100T
8GB RAM
128GB SSD Patriot
4 port Intel network card IBM/Intel
Been running for over a year without any issue.

[userunkn4wn]

First of all, thanks for all the replies. For now, the issue seems to be "resolved".

Here is what I noticed: if I boot with a monitor attached, unplug it, and then restart or shut down, the system hangs on the next boot. It seems like the system expects the monitor to still be there, though I might be wrong. Performing a forced shutdown resolves the issue on the following start.

This only occurs when booting with a monitor and shutting down without one. Hot-plugging the monitor while the system is running does not affect the boot process.

My previous confusion was likely caused by a mix of these scenarios, which made the behavior seem inconsistent and hard to reproduce reliably. In the end, I just wasn't connecting the dots regarding the system's prior state (booted with or without monitor).

This might be common knowledge, but I wanted to share my findings just in case it helps someone else. I still don't know the exact cause, but I at least have a workaround.

[nero355]

Yes, a display emulator would be a relatively easy but not a really satisfying fix, but yeah, I will consider this when I run out of options.

Just get one for a couple of bucks and be HAPPY! :)

I have a HDMI dummy attached to my good old Intel NUC 2820 FYKH and it works FLAWLESS!!!

#eBayStuffFTW :P

[drosophila]

Yes, a display emulator would be a relatively easy but not a really satisfying fix, but yeah, I will consider this when I run out of options.

Just get one for a couple of bucks and be HAPPY! :)

No, this still is a hard/firm/software bug and the emulator is a workaround with additional costs, not only in hardware but also in power draw and maintenance effort required. Since there are several different systems exhibiting the issue (maybe @Colt45 actually also didn't consider the previous connection state?), this would be either an EFI (immature, just like BIOS was 30 years ago), HDMI (especially HDCP is a PoS IMNSHO, waste of power, hassle and bugs such as these are pre-determined) or FreeBSD issue (which ultimately is due to the overly complex and contrived and ultimately worthless BS that the aforementioned schemes force upon OS developers, and these asinine b*tards demand licensing fees on top of all this even, and for what? So we cannot use our devices that we paid for the way they are meant to be used. Sorry for the rant.).

[userunkn4wn]

Unfortunately, I have to retract my statement regarding the monitor state conclusion.

The system still occasionally hangs during boot. As I gathered from the boot logs in /var/log/system/, when the system hangs, it doesn't even write to these logs, meaning the kernel is not loaded yet. It must be a bootloader issue. Unfortunately, I don't know how to log bootloader output (or if that's even possible), so it's not really possible to determine the root cause of these hangs.

I just ordered a few HDMI dummy plugs to make the system more reliable, as I cannot afford to have it go down after a power outage.

However, I am still interested in solving this problem at a fundamental level rather than relying on this band-aid fix.

Any suggestions are much welcome!

[Patrick M. Hausen]

Did you try setting the primary console to serial and the secondary to none as I already advised?

[userunkn4wn]

Did you try setting the primary console to serial and the secondary to none as I already advised?

Yes, I tried that setting as suggested, but unfortunately, it didn't solve the issue. I suspect this workaround only applies when the system fails to boot altogether. In my case, it's just inconsistent, it boots fine about 3 out of 4 times, but occasionally gets stuck.

That makes it really difficult to isolate the cause, as the behavior is fairly stochastic. Sometimes it works 5 to 6 times in a row, leading me to believe it's fixed, only for it to fail on the next attempt. It's tough to troubleshoot under these conditions.

[Patrick M. Hausen]

We need to fix that bug in the boot loader. I am quite a bit disappointed the issue in the tracker has not got any attention. In March a volunteer intending to fix the problem asked for some information - no response so far.

I'll raise some "management attention" in a week ;-)

Kind regards,
Patrick