Power loss recovery on Protectli V1410 - USB COM port & system time

[OPNenthu]

I don't know if this issue affects more units than just mine, or if this is maybe on my end (defective PC motherboard?), but I wanted to document the recovery steps.  I did forward my observations to Protectli as well.

Specs:
Vault V1410 (Intel N5105, 8GB) with Protectli coreboot version 0.9.3.  The coreboot detail is important because there's no option to set the RTC that I'm aware of, so will need to be done through the OS.

I don't know if this affects units with the stock AMI UEFI.

The issue:
Following an extended power loss event where the Vault remains disconnected from mains power or battery backup for some time (>1hr in my case), the USB COM port becomes inoperable and fails to list in Windows Device Manager or in the Linux /dev filesystem.  The firewall remains accessible only via GUI and SSH. The issue persists across power cycles.

(EDIT: I haven't tried the vga console as mine runs headless.)

Evidence:
You can observe that the device is not being recognized on your PC that is connected with the USB COM cable.  For example, in Linux 'dmesg' output there will not be any reference to the expected device (/dev/ttyUSB0) that is typically there.  Also there will be no such character device on the /dev on the filesystem:

Code Select Expand

$ sudo dmesg | grep tty
[    0.152646] printk: legacy console [tty0] enabled
[    0.791432] serial8250: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A


Code Select Expand

$ ls -l /dev/ttyUSB*
ls: cannot access '/dev/ttyUSB*': No such file or directory

On Windows there will be an error indicator in Device Manager.  The COM port will have become an unrecognized device that fails to initialize.

The fix:
1) Open the device and perform a CMOS reset by shorting two pins.  https://kb.protectli.com/kb/cmos-reset/

This will restore the serial port, but will also wipe out the system time.

2) DNS resolution may be blocked due to the time error and the NTP service in OPNsense will be unable to sync.  Manually set the date to the current wall clock time to within 1-2 minutes.  On the OPNsense console (as root):

$ date yymmddhhMM

For example, enter "2507301300" for "Jul 30 2025 at 1pm"

This will get NTP and other services unstuck, but may take a few minutes to sync.

Validation:

Code Select Expand

$ sudo dmesg | grep tty
[    0.152646] printk: legacy console [tty0] enabled
[    0.791432] serial8250: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 8856.625541] usb 1-1: FTDI USB Serial Device converter now attached to ttyUSB0     <--- this reappears


Code Select Expand

$ ls -l /dev/ttyUSB*
crw-rw---- 1 root dialout 188, 0 Jul 30 00:37 /dev/ttyUSB0

If on Windows, you should again see the COM0 port in Device Manager.


-----

I did see this issue multiple times already as we have been having more frequent power outages in our region, and also I've seen it on two separate V1410s.  I had gotten a replacement for my original one due to a different issue, but that one exhibited this problem as well. 

On the most recent occurrence I did happen to have a serial terminal session open when the power went out, so it could be a factor.  I don't recall if I was able to close the session cleanly and I don't remember if that was always the case the other times it happened.  If you own this device and depend on the USB COM then maybe consider keeping around a small screwdriver.

[OPNenthu]

Quick update from Protectli support:

- Possible dead CMOS battery causing loss of clock time and getting the USB COM port into a bad state when disconnected from power (some BIOS state issue, perhaps).  They offered a replacement battery.

- Unclear why a CMOS reset brings the COM port back up without replacing the battery.  More testing needed.  Protectli trying to reproduce, but have not so far seen this issue.

- I asked why it happened on two separate units which were both new (should have had good batteries).  It's unclear, but there are a couple thoughts on this:

  1) It's unlikely but not impossible that I got two weak batteries in a row.

  2) There could be an unidentified vampiric drain on the battery.

  As a side note, I mentioned to them that I keep the USB COM cable connected to the device 24/7, as it sits on my desk next to my PC.  Not sure if that's contributing.  I'm wondering if that keeps the FTDI chip active and if this has some kind of connection to the CMOS battery (not verified).

[cottec]

Super weird behaviour on my side as well.

This morning one single client didnt get an IP Adress.
The Protectli 1410 OPsense leased an IP but the client connected to the Wifi AP using a local 169.254 adress

Tried to reboot the Protecli in OPsene WebGUI.

It never booted again...

COM not working either.



Took out the Bios Battery 0.2Volts


Can a dead battery cause the whole system not to boot anymore?!

[OPNenthu]

Hi @cottec,

Not sure what's going on with yours specifically but the COM issue sounds familiar.  I have a newer production run of the V1410 now and these issues haven't come back.  I would reach out to Protectli support.

You might be able to get COM port responding again with a CMOS reset: https://kb.protectli.com/kb/cmos-reset/.  I remember this working for me.

Replace the battery first if you can, and make sure you are on the latest firmware.

[Nullman]

Can a dead battery cause the whole system not to boot anymore?!

Yes. Not only on Protectli devices. Modern motherboards show no signs of life if the battery goes below 2.5V. You would think 2.5V is not that bad, but that 2.5V becomes 0.9V when under load. So yes. Replace your battery.

[cottec]

Hi @cottec,

Not sure what's going on with yours specifically but the COM issue sounds familiar.  I have a newer production run of the V1410 now and these issues haven't come back.  I would reach out to Protectli support.

You might be able to get COM port responding again with a CMOS reset: https://kb.protectli.com/kb/cmos-reset/.  I remember this working for me.

Replace the battery first if you can, and make sure you are on the latest firmware.

as my emmc is dead and I'm barely within warranty I might still send it to them

did you try updating the NIC's firmware with the newer build?

Can a dead battery cause the whole system not to boot anymore?!

Yes. Not only on Protectli devices. Modern motherboards show no signs of life if the battery goes below 2.5V. You would think 2.5V is not that bad, but that 2.5V becomes 0.9V when under load. So yes. Replace your battery.

yeah will do that for sure, but I wasn't expecting such issues tbh. Seems to be a coreboot thing not to boot up at all with the battery empty

Still dont understand why that battery's dead after 2 years...

[Nullman]

Seems to be a coreboot thing not to boot up at all with the battery empty

It has nothing to do with coreboot. Modern motherboards are all AMI, and they behave exactly the same.

Still dont understand why that battery's dead after 2 years...

If your device is running hot all the time, that can accelerate battery degradation. Keep in mind that some Protecli devices dont use classic coin cell battery. They use the ones that are wrapped in shrink and have a short piece of wire with the connector. If thats the case, you need to pay attention when getting a new one because of polarity.

[OPNenthu]

as my emmc is dead and I'm barely within warranty I might still send it to them

did you try updating the NIC's firmware with the newer build?

I kept the NIC firmware as-is and honestly now with OPNsense 26.7.x I feel even less of a reason to change it.  Things seem to be behaving.

If they're going to ship you a new one anyway you might as well ask them if they can update it.

BTW, you probably saw my note above about setting the date manually in FreeBSD the first time you boot after a battery change / CMOS reset.  Coreboot doesn't let you set the system time unfortunately so until you do it in the OS you'll have problems getting NTP to sync.  FYI.

I agree w/ Nullman, these things do seem to go through batteries faster than the typical PC/laptops I'm used to.  However, I think the first production units (the "v1" if you will) did have some issue.

[pfry]

[...]Modern motherboards show no signs of life if the battery goes below 2.5V.[...]

I haven't had a problem with mine (a few Asrock, Asus, Gigabyte, and Supermicro boards). I've noticed that most eat batteries when powered off (5 year life at best), and one Asrock Rack board eats them in no time (perhaps a year). It boots fine, though - the only obvious symptom is the time (it has few meaningful configuration options).

I wish the bloody things used flash for configuration (active as opposed to backup, and some of mine are old enough to lack even that). A (convenient, supported) bigger battery option would be nice, too (e.g. 2 AA). Oh, context: I like to use systems for 10+ years these days. Advances in compute just aren't as exciting as they were 25+ years ago.