OPNsense Forum

English Forums => Hardware and Performance => Topic started by: OPNenthu on July 30, 2025, 10:28:08 AM

Title: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on July 30, 2025, 10:28:08 AM
I don't know if this issue affects more units than just mine, or if this is maybe on my end (defective PC motherboard?), but I wanted to document the recovery steps.  I did forward my observations to Protectli as well.

Specs:
Vault V1410 (Intel N5105, 8GB) with Protectli coreboot version 0.9.3.  The coreboot detail is important because there's no option to set the RTC that I'm aware of, so will need to be done through the OS.

I don't know if this affects units with the stock AMI UEFI.

The issue:
Following an extended power loss event where the Vault remains disconnected from mains power or battery backup for some time (>1hr in my case), the USB COM port becomes inoperable and fails to list in Windows Device Manager or in the Linux /dev filesystem.  The firewall remains accessible only via GUI and SSH. The issue persists across power cycles.

(EDIT: I haven't tried the vga console as mine runs headless.)

Evidence:
You can observe that the device is not being recognized on your PC that is connected with the USB COM cable.  For example, in Linux 'dmesg' output there will not be any reference to the expected device (/dev/ttyUSB0) that is typically there.  Also there will be no such character device on the /dev on the filesystem:

$ sudo dmesg | grep tty
[    0.152646] printk: legacy console [tty0] enabled
[    0.791432] serial8250: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A

$ ls -l /dev/ttyUSB*
ls: cannot access '/dev/ttyUSB*': No such file or directory

On Windows there will be an error indicator in Device Manager.  The COM port will have become an unrecognized device that fails to initialize.

The fix:
1) Open the device and perform a CMOS reset by shorting two pins.  https://kb.protectli.com/kb/cmos-reset/

This will restore the serial port, but will also wipe out the system time.

2) DNS resolution may be blocked due to the time error and the NTP service in OPNsense will be unable to sync.  Manually set the date to the current wall clock time to within 1-2 minutes.  On the OPNsense console (as root):

$ date yymmddhhMM

For example, enter "2507301300" for "Jul 30 2025 at 1pm"

This will get NTP and other services unstuck, but may take a few minutes to sync.

Validation:

$ sudo dmesg | grep tty
[    0.152646] printk: legacy console [tty0] enabled
[    0.791432] serial8250: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[ 8856.625541] usb 1-1: FTDI USB Serial Device converter now attached to ttyUSB0     <--- this reappears

$ ls -l /dev/ttyUSB*
crw-rw---- 1 root dialout 188, 0 Jul 30 00:37 /dev/ttyUSB0

If on Windows, you should again see the COM0 port in Device Manager.


-----

I did see this issue multiple times already as we have been having more frequent power outages in our region, and also I've seen it on two separate V1410s.  I had gotten a replacement for my original one due to a different issue, but that one exhibited this problem as well. 

On the most recent occurrence I did happen to have a serial terminal session open when the power went out, so it could be a factor.  I don't recall if I was able to close the session cleanly and I don't remember if that was always the case the other times it happened.  If you own this device and depend on the USB COM then maybe consider keeping around a small screwdriver.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on July 31, 2025, 07:10:39 PM
Quick update from Protectli support:

- Possible dead CMOS battery causing loss of clock time and getting the USB COM port into a bad state when disconnected from power (some BIOS state issue, perhaps).  They offered a replacement battery.

- Unclear why a CMOS reset brings the COM port back up without replacing the battery.  More testing needed.  Protectli trying to reproduce, but have not so far seen this issue.

- I asked why it happened on two separate units which were both new (should have had good batteries).  It's unclear, but there are a couple thoughts on this:

  1) It's unlikely but not impossible that I got two weak batteries in a row.

  2) There could be an unidentified vampiric drain on the battery.

  As a side note, I mentioned to them that I keep the USB COM cable connected to the device 24/7, as it sits on my desk next to my PC.  Not sure if that's contributing.  I'm wondering if that keeps the FTDI chip active and if this has some kind of connection to the CMOS battery (not verified).


Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: cottec on September 07, 2026, 11:17:39 AM
Super weird behaviour on my side as well.

This morning one single client didnt get an IP Adress.
The Protectli 1410 OPsense leased an IP but the client connected to the Wifi AP using a local 169.254 adress

Tried to reboot the Protecli in OPsene WebGUI.

It never booted again...

COM not working either.



Took out the Bios Battery 0.2Volts


Can a dead battery cause the whole system not to boot anymore?!
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on September 07, 2026, 10:28:49 PM
Hi @cottec,

Not sure what's going on with yours specifically but the COM issue sounds familiar.  I have a newer production run of the V1410 now and these issues haven't come back.  I would reach out to Protectli support.

You might be able to get COM port responding again with a CMOS reset: https://kb.protectli.com/kb/cmos-reset/.  I remember this working for me.

Replace the battery first if you can, and make sure you are on the latest firmware.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: Nullman on September 08, 2026, 02:05:29 PM
Quote from: cottec on September 07, 2026, 11:17:39 AMCan a dead battery cause the whole system not to boot anymore?!

Yes. Not only on Protectli devices. Modern motherboards show no signs of life if the battery goes below 2.5V. You would think 2.5V is not that bad, but that 2.5V becomes 0.9V when under load. So yes. Replace your battery.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: cottec on September 08, 2026, 02:11:49 PM
Quote from: OPNenthu on September 07, 2026, 10:28:49 PMHi @cottec,

Not sure what's going on with yours specifically but the COM issue sounds familiar.  I have a newer production run of the V1410 now and these issues haven't come back.  I would reach out to Protectli support.

You might be able to get COM port responding again with a CMOS reset: https://kb.protectli.com/kb/cmos-reset/.  I remember this working for me.

Replace the battery first if you can, and make sure you are on the latest firmware.


as my emmc is dead and I'm barely within warranty I might still send it to them

did you try updating the NIC's firmware with the newer build?


Quote from: Nullman on September 08, 2026, 02:05:29 PM
Quote from: cottec on September 07, 2026, 11:17:39 AMCan a dead battery cause the whole system not to boot anymore?!

Yes. Not only on Protectli devices. Modern motherboards show no signs of life if the battery goes below 2.5V. You would think 2.5V is not that bad, but that 2.5V becomes 0.9V when under load. So yes. Replace your battery.
yeah will do that for sure, but I wasn't expecting such issues tbh. Seems to be a coreboot thing not to boot up at all with the battery empty

Still dont understand why that battery's dead after 2 years...
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: Nullman on September 08, 2026, 02:19:04 PM
Quote from: cottec on September 08, 2026, 02:11:49 PMSeems to be a coreboot thing not to boot up at all with the battery empty

It has nothing to do with coreboot. Modern motherboards are all AMI, and they behave exactly the same.

Quote from: cottec on September 08, 2026, 02:11:49 PMStill dont understand why that battery's dead after 2 years...
If your device is running hot all the time, that can accelerate battery degradation. Keep in mind that some Protecli devices dont use classic coin cell battery. They use the ones that are wrapped in shrink and have a short piece of wire with the connector. If thats the case, you need to pay attention when getting a new one because of polarity.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on September 08, 2026, 07:58:10 PM
Quote from: cottec on September 08, 2026, 02:11:49 PMas my emmc is dead and I'm barely within warranty I might still send it to them

did you try updating the NIC's firmware with the newer build?

I kept the NIC firmware as-is and honestly now with OPNsense 26.7.x I feel even less of a reason to change it.  Things seem to be behaving.

If they're going to ship you a new one anyway you might as well ask them if they can update it.

BTW, you probably saw my note above about setting the date manually in FreeBSD the first time you boot after a battery change / CMOS reset.  Coreboot doesn't let you set the system time unfortunately so until you do it in the OS you'll have problems getting NTP to sync.  FYI.

I agree w/ Nullman, these things do seem to go through batteries faster than the typical PC/laptops I'm used to.  However, I think the first production units (the "v1" if you will) did have some issue.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: pfry on September 08, 2026, 09:29:09 PM
Quote from: Nullman on September 08, 2026, 02:05:29 PM[...]Modern motherboards show no signs of life if the battery goes below 2.5V.[...]

I haven't had a problem with mine (a few Asrock, Asus, Gigabyte, and Supermicro boards). I've noticed that most eat batteries when powered off (5 year life at best), and one Asrock Rack board eats them in no time (perhaps a year). It boots fine, though - the only obvious symptom is the time (it has few meaningful configuration options).

I wish the bloody things used flash for configuration (active as opposed to backup, and some of mine are old enough to lack even that). A (convenient, supported) bigger battery option would be nice, too (e.g. 2 AA). Oh, context: I like to use systems for 10+ years these days. Advances in compute just aren't as exciting as they were 25+ years ago.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: cottec on September 10, 2026, 10:35:47 PM
Quote from: OPNenthu on September 08, 2026, 07:58:10 PMBTW, you probably saw my note above about setting the date manually in FreeBSD the first time you boot after a battery change / CMOS reset.  Coreboot doesn't let you set the system time unfortunately so until you do it in the OS you'll have problems getting NTP to sync.  FYI.
I waited for a new battery to arrive but it's still dead, will send it to them tomorrow.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: nero355 on September 14, 2026, 12:27:12 AM
Quote from: pfry on September 08, 2026, 09:29:09 PMA (convenient, supported) bigger battery option would be nice, too (e.g. 2 AA).
Rechargeable AA batteries can be Low Self Discharge models for many years now : Why aren't these things too ?! :(

System ON = Charge the battery to 100% and cut off when full.
System OFF = Use the battery for the CMOS Clock until next boot.

Maybe it will be that simple one day... Who knows...


QuoteOh, context: I like to use systems for 10+ years these days. Advances in computer hardware just aren't as exciting as they were 25+ years ago.
+1 here :)
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on October 02, 2026, 03:19:55 AM
Incidentally, mine is now again failing to enumerate the USB serial console on the admin PC (currently a Linux box).

Screenshot from 2026-10-01 20-40-47.png

In 'dmesg' there's some USB related error:

[  158.544011] usb 1-1: new full-speed USB device number 12 using xhci_hcd
[  158.748063] usb 1-1: device descriptor read/64, error -71
[  159.032069] usb 1-1: device descriptor read/64, error -71
[  159.313023] usb 1-1: new full-speed USB device number 13 using xhci_hcd
[  159.517076] usb 1-1: device descriptor read/64, error -71
[  159.800081] usb 1-1: device descriptor read/64, error -71
[  159.905782] usb usb1-port1: attempt power cycle
[  160.354040] usb 1-1: new full-speed USB device number 14 using xhci_hcd
[  160.380898] usb 1-1: Device not responding to setup address.
[  160.616897] usb 1-1: Device not responding to setup address.
[  160.822041] usb 1-1: device not accepting address 14, error -71
[  160.822140] usb 1-1: WARN: invalid context state for evaluate context command.
[  161.001048] usb 1-1: new full-speed USB device number 15 using xhci_hcd
[  161.027898] usb 1-1: Device not responding to setup address.
[  161.264898] usb 1-1: Device not responding to setup address.
[  161.470051] usb 1-1: device not accepting address 15, error -71
[  161.470150] usb 1-1: WARN: invalid context state for evaluate context command.
[  161.473864] usb usb1-port1: unable to enumerate USB device

On the OPNsense side, 'dmesg' shows that the console is up.  The 'FCR is broken' part was always there, so ignoring that.

[1] uart: ns8250: UART FCR is broken (0x1)
[1] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
[1] uart0: console (115200,n,8,1)

Anything else I can check in OPNsense?  The serial console is already enabled and set as primary in admin settings.  Haven't changed anything there.

I hope it's just the battery again and ordered a couple to try.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on October 08, 2026, 05:05:11 PM
Can now confirm: the serial console comes up again immediately after battery change.  I'm left with some thoughts about this after having experienced it twice on two different units (same model):

1) The CR1220 battery that it uses is quite small relative to the typical CMOS batteries e.g. CR2025/2032.  It has only <40 mAh capacity, whereas the other types have 160-225+.

2) I think the embedded USB/serial converter must be powered from it as it's the first thing to go.

I think I'm comfortable to say that if you own one of these you should keep a spare battery and best to replace it annually, regardless.  Don't keep the USB serial cable connected to the port 24/7 like I do because even if the terminal session is closed the chip is apparently still active.

Quote from: cottec on September 10, 2026, 10:35:47 PMI waited for a new battery to arrive but it's still dead, will send it to them tomorrow.

Hope you had some resolution on this.

Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: nero355 on October 08, 2026, 08:25:07 PM
Quote from: OPNenthu on October 08, 2026, 05:05:11 PM2) I think the embedded USB/serial converter must be powered from it as it's the first thing to go.
Would it be an option to have something like this : https://www.aten.com/global/en/products/usb-solutions/converters/uc232a/
Attached 24/7 to it as the backup of the onboard Serial Console ?!
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on October 08, 2026, 10:57:09 PM
Quote from: nero355 on October 08, 2026, 08:25:07 PMWould it be an option to have something like this : https://www.aten.com/global/en/products/usb-solutions/converters/uc232a/
Attached 24/7 to it as the backup of the onboard Serial Console ?!

No RS-232 port :-/

Just a USB-C interface to an FTDI chip.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: nero355 on October 09, 2026, 12:18:00 AM
Quote from: OPNenthu on October 08, 2026, 10:57:09 PMNo RS-232 port :-/
As in you don't have one ?

You can use the same thing to connect to it, but you need the right cable between them ofcourse :)

QuoteJust a USB-C interface to an FTDI chip.
Is what you are using to connect to the Protectli or the Serial Port on it is using it ?
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on October 09, 2026, 12:38:07 AM
There's no RS-232 serial port on the Protectli.  It has a USB-C port labeled "COM" which is internally wired to the FTDI UART and can only be used for that purpose:

https://kb.protectli.com/kb/v1000-hardware-overview/

They supply a USB cable with the unit.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: Nullman on October 09, 2026, 06:44:15 AM
Quote from: OPNenthu on October 08, 2026, 05:05:11 PMDon't keep the USB serial cable connected to the port 24/7 like I do because even if the terminal session is closed the chip is apparently still active.
You should never keep anything you dont use attached to your USB ports. Your serial console cable is active device because it acts like a USB to RS232 adapter. As soon as you plug it into your USB-C port, USB controller becomes active. It doesnt matter if you are using it or not. Rest assured It is not powered by CMOS battery. Thats not how it works. However, if it stays active for a long time it can get really hot. On some Protecli devices, CMOS battery is glued on metallic heat sink / shielding part where ports or USB controllers are. If the battery is continuously exposed to that heat, it will die prematurely. You learned that the hard way.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on October 09, 2026, 03:21:18 PM
I didn't consider USB controllers being active and keeping the battery hot.  In that case, I'm not suprised- I keep a UPS and an external fan plugged into it as well.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: Nullman on October 09, 2026, 04:33:32 PM
Quote from: OPNenthu on October 09, 2026, 03:21:18 PMI didn't consider USB controllers being active and keeping the battery hot.  In that case, I'm not suprised

Batteries dont like heat. Especially CR1220 batteries because they are smaller, and they soak heat much easier and die much quicker. This is also one of many reasons why you should never use USB>LAN cards as primary or secondary WAN/LAN interface even if its supported by FreeBSD. USB ports should only be used temporarily for console access or installing OS via USB stick. As soon as you are done, remove everything you dont need.

Quote from: OPNenthu on October 09, 2026, 03:21:18 PMI keep a UPS and an external fan plugged into it as well.

This is the worst idea ever. Its convenient, yes, but you are putting a lot of stress on whole device by doing this. Thats not what it was designed for even if it works. Fans should have their own power supply. This way you avoid interference that fan motors can cause and you do not stress your USB ports. Just because you can, doesnt mean you should.

Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: OPNenthu on October 09, 2026, 04:52:13 PM
Why do the regular USB ports remain functional when the CMOS battery gets low but only the serial console port stops working?

I think that even if the FTDI chip doesn't directly use the battery, something else related to the COM port clearly depends on it.
Title: Re: Power loss recovery on Protectli V1410 - USB COM port & system time
Post by: Nullman on October 09, 2026, 05:30:28 PM
Quote from: OPNenthu on October 09, 2026, 04:52:13 PMWhy do the regular USB ports remain functional when the CMOS battery gets low but only the serial console port stops working?

I think that even if the FTDI chip doesn't directly use the battery, something else related to the COM port clearly depends on it.

That i dont know.