22.1 crash my WatchGuard XTM-870 –> Apei boot problem with FreeBSD 13

[estragon]

For a few years now, I have been using second-hand routers where I install a **sense in them, (first PFsense, then for some time OPNsense). It opens up possibilities for me that I could not fund otherwise. A big thank you to the whole OPNsense team and to all those who have gone before, and to the whole BSD movement for this superb software and knowledge.

My last machines are WatchGuard XTM-870 and XTM-1525 (in fact it is a derivative of the Lanner FW-8758) machines a little old, but very reliable, with very good characteristics and powerful (for my needs in all cases and it is possible to find used ones at very affordable prices, since they are a bit dated!)

The machine was updated regularly without any problems. But the last 22.1 update broke everything for me (by the way, a little reminder in passing: don't forget the backups!! I hadn't done it and it was complicated to recover the configuration file, but that's is another story and above all just my fault!)

After the crash, wanting to install a clean 22.1, I couldn't do it, while I could easily put a clean 21.7 back. This put me on the trail of changing the operating system. In fact 21 is based on a derivative of FreeBSD 11 while 22.1 is based on a much more recent true FreeBSD.
After several hours of testing, I was able to define that all FreeBSDs up to 12.1 can be installed without problem on this machine, but not the following ones (from 12.2 and all following impossible!)

After a lot of reading on the forums and IRC and especially the precious help of several people, the problem was the APEI management (part of ACPI). I also managed to inform the maintainer of this module on FreeBSD, Alexander, whom I thank very much.

By using the following variable at boot time I was able to work around the problem :

set hint.apei.0.disabled=1

I have a few questions:

1/ Have other users of WatchGuard or other machines encountered this problem? Were they able to update the BIOS (I couldn't). Where did they find another workaround?

2 / having no computer studies base, I find it difficult for me to better define the problem and to be able to solve it, but perhaps others have experience that would allow things to move forward. It is happy that I can do tests on my machine, if that can help to solve the issue and improve the product for everyone.

3/ while waiting for a more lasting solution to be found, I need to be sure that future updates will not modify my workaround (and broke all an other time). From what I have read the most consistent create a file /boot/loader.conf.local and put hint.apei.0.disabled=1 there. I can consider to do in ssh, but i prefer to make it from GUI interface if possible, to be more in the spirit of OPNsense.
I go on the page system:settings:tunables but I'm having trouble to create the file loader.conf.local. Could you give me a hand to do it as correctly as possible.

Thank you and have a nice day (and sorry for my googelized English!)

[magnust]

My Intel Core i5-7200U China fw-box that has been running flawlessly for years started randomly rebooting every few hours right after upgrading to v22.1.

I added your tip, set hint.apei.0.disabled=1 and it hasn't crashed yet (9 hours).

I am IMMENSLY grateful for your work in finding this workaround!!!!!

I added it to the settings under System -> Settings -> Tunables which makes it stick (AFAIK this setting will even be restored during a from scratch install if using a backup configuration file to restore all settings)


Thank you!!!!!!!

 

[magnust]

Odd thing, my box has crashed a few times since I was here last but nowhere near as often as it did before applying this to tunables. 

[estragon]

I was happy that my experience was able to help you, but a little above that it did not work as hoped!

Since I have 2 boxes, I put one in the main router, and with the trick of the 1st post I was able to install a new version 22.1 which works well (for the moment, but it's been a few days already!). And with the other machine I did some tests, and I installed different versions of Freebsd ... or rather I couldn't install them!
After having exchanged with Alexander Motin (who takes care of the ACPI part of Freebsd) he modified some code from the CURRENT version and it works in my case.
And if all goes well, it should then go back to future versions of OPNsense (if I understood correctly)

If you think you can do the test with freebsd I would gladly send you the links.
On the other hand, I am not strong enough to be able to help you more in the state of things. Maybe you have logs to better understand if it's the same problem?
I hope you will be able to solve your problem. For information, I had found a lot of help here but also on the IRC

[magnust]

It just went down and rebooted again. Was up aprox 24 hours this time. Thanks for the offer for links to testing but I only have one box with this hardware and no other machine to run as active FW so can't run test installs on it. I'm really 100% lost where to start, where and what to look for. If all else fails I might need to buy new hardware...

I'll take a look at the BIOS settings to see if I can see anything that might be worth trying out. Here's the dmesg for the first boot after the latest crash and reboot if anyone is curious.

Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-STABLE stable/22.1-n248053-232cb14f501 SMP amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(efifb): resolution 800x600
CPU: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (2700.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x806e9  Family=0x6  Model=0x8e  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0xc000000<IBPB,STIBP>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3999739904 (3814 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 1 2 3
wlan: mac acl policy registered
random: entropy device external interface
kbd0 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
acpi0: <ALASKA A M I >
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
xhci0: <Intel Sunrise Point-LP USB 3.0 controller> mem 0xdf610000-0xdf61ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <simple comms> at device 22.0 (no driver attached)
ahci0: <Intel Sunrise Point-LP AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xdf628000-0xdf629fff,0xdf62f000-0xdf62f0ff,0xdf62e000-0xdf62e7ff irq 16 at device 23.0 on pci0
ahci0: AHCI v1.31 with 3 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) I211 (Copper)> port 0xe000-0xe01f mem 0xdf500000-0xdf51ffff,0xdf520000-0xdf523fff irq 16 at device 0.0 on pci1
igb0: NVM V0.6 imgtype1
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 2 RX queues 2 TX queues
igb0: Using MSI-X interrupts with 3 vectors
igb0: Ethernet address: 40:62:31:02:cd:69
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
igb1: <Intel(R) I211 (Copper)> port 0xd000-0xd01f mem 0xdf400000-0xdf41ffff,0xdf420000-0xdf423fff irq 17 at device 0.0 on pci2
igb1: NVM V0.6 imgtype1
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 2 RX queues 2 TX queues
igb1: Using MSI-X interrupts with 3 vectors
igb1: Ethernet address: 40:62:31:02:cd:6a
igb1: netmap queues/slots: TX 2/1024, RX 2/1024
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
igb2: <Intel(R) I211 (Copper)> port 0xc000-0xc01f mem 0xdf300000-0xdf31ffff,0xdf320000-0xdf323fff irq 18 at device 0.0 on pci3
igb2: NVM V0.6 imgtype1
igb2: Using 1024 TX descriptors and 1024 RX descriptors
igb2: Using 2 RX queues 2 TX queues
igb2: Using MSI-X interrupts with 3 vectors
igb2: Ethernet address: 40:62:31:02:cd:6b
igb2: netmap queues/slots: TX 2/1024, RX 2/1024
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
igb3: <Intel(R) I211 (Copper)> port 0xb000-0xb01f mem 0xdf200000-0xdf21ffff,0xdf220000-0xdf223fff irq 19 at device 0.0 on pci4
igb3: NVM V0.6 imgtype1
igb3: Using 1024 TX descriptors and 1024 RX descriptors
igb3: Using 2 RX queues 2 TX queues
igb3: Using MSI-X interrupts with 3 vectors
igb3: Ethernet address: 40:62:31:02:cd:6c
igb3: netmap queues/slots: TX 2/1024, RX 2/1024
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib5
igb4: <Intel(R) I211 (Copper)> port 0xa000-0xa01f mem 0xdf100000-0xdf11ffff,0xdf120000-0xdf123fff irq 16 at device 0.0 on pci5
igb4: NVM V0.6 imgtype1
igb4: Using 1024 TX descriptors and 1024 RX descriptors
igb4: Using 2 RX queues 2 TX queues
igb4: Using MSI-X interrupts with 3 vectors
igb4: Ethernet address: 40:62:31:02:cd:6d
igb4: netmap queues/slots: TX 2/1024, RX 2/1024
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
igb5: <Intel(R) I211 (Copper)> port 0x9000-0x901f mem 0xdf000000-0xdf01ffff,0xdf020000-0xdf023fff irq 17 at device 0.0 on pci6
igb5: NVM V0.6 imgtype1
igb5: Using 1024 TX descriptors and 1024 RX descriptors
igb5: Using 2 RX queues 2 TX queues
igb5: Using MSI-X interrupts with 3 vectors
igb5: Ethernet address: 40:62:31:02:cd:6e
igb5: netmap queues/slots: TX 2/1024, RX 2/1024
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <memory> at device 31.2 (no driver attached)
hdac0: <Intel Kaby Lake-LP HDA Controller> mem 0xdf620000-0xdf623fff,0xdf600000-0xdf60ffff irq 16 at device 31.3 on pci0
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
Timecounter "TSC-low" frequency 1356001087 Hz quality 1000
Timecounters tick every 1.000 msec
hdacc0: <Intel Kaby Lake HDA CODEC> at cad 2 on hdac0
hdaa0: <Intel Kaby Lake Audio Function Group> at nid 1 on hdacc0
pcm0: <Intel Kaby Lake (HDMI/DP 8ch)> at nid 3 on hdaa0
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Root mount waiting for: usbus0 CAM
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <TS32GMSA370 P1225CH1> ACS-2 ATA SATA 3.x device
ada0: Serial Number G100930197
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 30533MB (62533296 512 byte sectors)
uhub0: 18 ports with 18 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <Logitech USB Receiver> at usbus0
ukbd0 on uhub0
ukbd0: <Logitech USB Receiver, class 0/0, rev 2.00/12.01, addr 1> on usbus0
kbd1 at ukbd0
mountroot: waiting for device /dev/gpt/rootfs...
WARNING: /mnt was not properly dismounted
igb0: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80001b2d000), ignoring.
igb1: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb1 (ifp 0xfffff800017bd000), ignoring.
igb2: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb2 (ifp 0xfffff80001930800), ignoring.
igb3: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb3 (ifp 0xfffff80001b73800), ignoring.
igb4: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb4 (ifp 0xfffff80001b75800), ignoring.
igb5: link state changed to UP
debugnet_any_ifnet_update: Bad dn_init result from igb5 (ifp 0xfffff80001bc6800), ignoring.
pchtherm0: <Skylake PCH Thermal Subsystem> mem 0xdf633000-0xdf633fff irq 18 at device 20.2 on pci0
ig4iic0: <Intel Sunrise Point-LP I2C Controller-0> mem 0xdf632000-0xdf632fff irq 16 at device 21.0 on pci0
ig4iic0: Using MSI
iicbus0: <Philips I2C bus (ACPI-hinted)> on ig4iic0
ig4iic1: <Intel Sunrise Point-LP I2C Controller-1> mem 0xdf631000-0xdf631fff irq 17 at device 21.1 on pci0
ig4iic1: Using MSI
iicbus1: <Philips I2C bus (ACPI-hinted)> on ig4iic1
ichsmb0: <Intel Sunrise Point-LP SMBus controller> port 0xf040-0xf05f mem 0xdf62a000-0xdf62a0ff irq 16 at device 31.4 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_wmi0: <ACPI-WMI mapping> on acpi0
acpi_wmi0: cannot find EC device
ums0 on uhub0
ums0: <Logitech USB Receiver, class 0/0, rev 2.00/12.01, addr 1> on usbus0
ums0: 16 buttons and [XYZT] coordinates ID=2
uhid0 on uhub0
uhid0: <Logitech USB Receiver, class 0/0, rev 2.00/12.01, addr 1> on usbus0
lo0: link state changed to UP
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
tun2: changing name to 'ovpns2'
tun1: changing name to 'ovpns1'
igb4: link state changed to DOWN
igb3: link state changed to DOWN
igb2: link state changed to DOWN
igb5: link state changed to DOWN
igb0: link state changed to DOWN
igb3: link state changed to UP
igb4: link state changed to UP
igb0: link state changed to UP
igb0: link state changed to DOWN
igb2: link state changed to UP
igb5: link state changed to UP
igb1: link state changed to DOWN
igb0: link state changed to UP
igb1: link state changed to UP
arp: (comment: ip number removed) moved from 40:62:31:02:cd:6a to 00:00:5e:00:01:0a on igb0
pflog0: permanently promiscuous mode enabled
ovpns2: link state changed to UP
ovpns1: link state changed to UP
arp:  (comment: ip number removed) moved from 00:00:5e:00:01:0a to 40:62:31:02:cd:6a on igb0
ovpns2: link state changed to DOWN
652.531013 [ 849] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
652.637411 [ 849] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
ovpns2: link state changed to UP
igb0: link state changed to DOWN
652.970086 [ 849] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
653.363011 [ 849] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
653.364705 [ 849] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
igb1: link state changed to DOWN
653.625620 [ 849] iflib_netmap_config       txr 2 rxr 2 txd 1024 rxd 1024 rbufsz 2048
ovpns1: link state changed to DOWN
ovpns1: link state changed to UP
igb0: link state changed to UP
igb1: link state changed to UP
ovpns2: link state changed to DOWN
ovpns2: link state changed to UP
ovpns1: link state changed to DOWN
ovpns1: link state changed to UP