Frequent (almost daily) crashing on the primary FW

[javier.wong]

Hi guys,

I have a clustered pair of FWs. Some months back the primary firewall started to crash frequently almost on a daily basis. The secondary FW does not crash at all. The primary firewall is on a Dell server whereas the secondary one is on a low end network appliance. I have tried the following to no avail:
- Reformatted the FW and reloaded the configuration
- Changed the RAM DIMMS
- Upgraded to the latest version 19.7.8

Does anyone have any idea what may be causing this?

Thanks in advance.

Here is the crash log (had to remove the top portion due to the message limit):

Code: [Select]

Tracing command devd pid 99454 tid 100121 td 0xfffff80006269000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe02380d9720
mi_switch() at mi_switch+0xe6/frame 0xfffffe02380d9750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe02380d97b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe02380d97f0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x18c/frame 0xfffffe02380d9850
seltdwait() at seltdwait+0x7b/frame 0xfffffe02380d9890
kern_select() at kern_select+0x850/frame 0xfffffe02380d9a80
sys_select() at sys_select+0x56/frame 0xfffffe02380d9ac0
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe02380d9bf0
fast_syscall_common() at fast_syscall_common+0x101/frconfig.txt0600001103513604326477  7547 ustarrootwheeloptions   CONFIG_AUTOGENERATED
ident   SMP
machine   amd64
cpu   HAMMER
makeoptions   WITH_CTF=1
options   PAX_JAIL_SUPPORT
options   PAX_CONTROL_EXTATTR
options   PAX_CONTROL_ACL_OVERRIDE_SUPPORT
options   PAX_CONTROL_ACL
options   PAX_INSECURE_MODE
options   PAX_SEGVGUARD
options   PAX_HARDENING
options   PAX_ASLR
options   PAX
options   TCP_SIGNATURE
options   PPS_SYNC
options   MROUTING
options   IPSTEALTH
options   IPFIREWALL_VERBOSE
options   IPFIREWALL_DEFAULT_TO_ACCEPT
options   GEOM_UZIP
options   GEOM_MIRROR
options   GEOM_ELI
options   GEOM_BDE
options   DDB
options   XENHVM
options   USB_DEBUG
options   ATH_ENABLE_11N
options   AH_AR5416_INTERRUPT_MITIGATION
options   AH_SUPPORT_AR5416
options   IEEE80211_SUPPORT_MESH
options   IEEE80211_AMPDU_AGE
options   IEEE80211_DEBUG
options   SC_PIXEL_MODE
options   VESA
options   AHD_REG_PRETTY_PRINT
options   AHC_REG_PRETTY_PRINT
options   PCI_IOV
options   PCI_HP
options   ACPI_DMAR
options   EARLY_AP_STARTUP
options   DEVICE_NUMA
options   SMP
options   KDB_TRACE
options   KDB
options   RCTL
options   RACCT_DEFAULT_TO_DISABLED
options   RACCT
options   INCLUDE_CONFIG_FILE
options   DDB_CTF
options   KDTRACE_HOOKS
options   KDTRACE_FRAME
options   MAC
options   CAPABILITIES
options   CAPABILITY_MODE
options   AUDIT
options   HWPMC_HOOKS
options   KBD_INSTALL_CDEV
options   PRINTF_BUFR_SIZE=128
options   _KPOSIX_PRIORITY_SCHEDULING
options   SYSVSEM
options   SYSVMSG
options   SYSVSHM
options   STACK
options   KTRACE
options   SCSI_DELAY=5000
options   COMPAT_FREEBSD10
options   COMPAT_FREEBSD9
options   COMPAT_FREEBSD7
options   COMPAT_FREEBSD6
options   COMPAT_FREEBSD5
options   COMPAT_FREEBSD4
options   COMPAT_FREEBSD32
options   GEOM_LABEL
options   GEOM_RAID
options   GEOM_PART_GPT
options   PSEUDOFS
options   PROCFS
options   CD9660
options   MSDOSFS
options   NFS_ROOT
options   NFSLOCKD
options   NFSD
options   NFSCL
options   MD_ROOT
options   QUOTA
options   UFS_GJOURNAL
options   UFS_DIRHASH
options   UFS_ACL
options   SOFTUPDATES
options   FFS
options   SCTP
options   TCP_OFFLOAD
options   IPSEC_SUPPORT
options   IPSEC
options   INET6
options   INET
options   PREEMPTION
options   SCHED_ULE
options   NEW_PCIB
options   GEOM_PART_MBR
options   GEOM_PART_EBR_COMPAT
options   GEOM_PART_EBR
options   GEOM_PART_BSD
device   isa
device   mem
device   io
device   uart_ns8250
device   cpufreq
device   acpi
device   pci
device   fdc
device   ahci
device   ata
device   mvs
device   siis
device   ahc
device   ahd
device   esp
device   hptiop
device   isp
device   mpt
device   mps
device   mpr
device   sym
device   trm
device   adv
device   adw
device   aic
device   bt
device   isci
device   ocs_fc
device   scbus
device   ch
device   da
device   sa
device   cd
device   pass
device   ses
device   amr
device   arcmsr
device   ciss
device   dpt
device   hptmv
device   hptnr
device   hptrr
device   hpt27xx
device   iir
device   ips
device   mly
device   twa
device   smartpqi
device   tws
device   aac
device   aacp
device   aacraid
device   ida
device   mfi
device   mlx
device   mrsas
device   pmspcv
device   twe
device   nvme
device   nvd
device   atkbdc
device   atkbd
device   psm
device   kbdmux
device   vga
device   splash
device   sc
device   vt
device   vt_vga
device   vt_efifb
device   agp
device   cbb
device   pccard
device   cardbus
device   uart
device   ppc
device   ppbus
device   lpt
device   ppi
device   puc
device   bxe
device   de
device   em
device   igb
device   ix
device   ixv
device   ixl
device   ixlv
device   le
device   ti
device   txp
device   vx
device   miibus
device   ae
device   age
device   alc
device   ale
device   bce
device   bfe
device   bge
device   cas
device   dc
device   et
device   fxp
device   gem
device   hme
device   jme
device   lge
device   msk
device   nfe
device   nge
device   pcn
device   re
device   rl
device   sf
device   sge
device   sis
device   sk
device   ste
device   stge
device   tl
device   tx
device   vge
device   vr
device   wb
device   xl
device   wlan
device   wlan_wep
device   wlan_ccmp
device   wlan_tkip
device   wlan_amrr
device   an
device   ath
device   ath_pci
device   ath_hal
device   ath_rate_sample
device   ipw
device   iwi
device   iwn
device   malo
device   mwl
device   ral
device   wi
device   wpi
device   loop
device   random
device   padlock_rng
device   rdrand_rng
device   ether
device   vlan
device   tun
device   md
device   gif
device   firmware
device   bpf
device   uhci
device   ohci
device   ehci
device   xhci
device   usb
device   ukbd
device   umass
device   sound
device   snd_cmi
device   snd_csa
device   snd_emu10kx
device   snd_es137x
device   snd_hda
device   snd_ich
device   snd_via8233
device   mmc
device   mmcsd
device   sdhci
device   virtio
device   virtio_pci
device   vtnet
device   virtio_blk
device   virtio_scsi
device   virtio_balloon
device   hyperv
device   xenpci
device   vmx
device   netmap
device   crypto
device   bwi
device   bwn
device   rum
device   run
device   siba_bwn
device   u3g
device   uark
device   uath
device   uftdi
device   umct
device   umodem
device   upgt
device   uplcom
device   ural
device   urtw
device   uvisor
device   uvscom
device   zyd
device   wlan_acl
device   wlan_xauth
device   speaker
msgbuf.txt0600005645713604326477  7606 ustarrootwheelCopyright (c) 2013-2018 The HardenedBSD Project.
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-RELEASE-p16-HBSD  87a7fc985c3(stable/19.7) amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
HardenedBSD: initialize and check features (__HardenedBSD_version 1100056 __FreeBSD_version 1102000).
CPU: Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz (2261.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106a5  Family=0x6  Model=0x1a  Stepping=5
  Features=0xbfebfbff
  Features2=0x9ce3bd
  AMD Features=0x28000800
  AMD Features2=0x1
  VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8244363264 (7862 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table:
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
ioapic1: Changing APIC ID to 1
ioapic0  irqs 0-23 on motherboard
ioapic1  irqs 32-55 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1130522594 Hz quality 1000
random: entropy device external interface
wlan: mac acl policy registered
000.000017 [4213] netmap_init               netmap: loaded module
kbd1 at kbdmux0
nexus0
cryptosoft0:  on motherboard
acpi0:  on motherboard
acpi0: Power Button (fixed)
cpu0:  on acpi0
cpu1:  on acpi0
cpu2:  on acpi0
cpu3:  on acpi0
atrtc0:  port 0x70-0x7f irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0:  port 0x40-0x5f irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pcib0: _OSC returned error 0x10
pci0:  on pcib0
pcib1:  at device 1.0 on pci0
pci1:  on pcib1
bce0:  mem 0xda000000-0xdbffffff irq 36 at device 0.0 on pci1
miibus0:  on bce0
brgphy0:  PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Using defaults for TSO: 65518/35/2048
bce0: Ethernet address: 00:26:b9:30:42:12
bce0:
<5>bce0: link state changed to DOWN
ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (4.6.4); Bufs (RX:2;TX:2;PG:; Flags (SPLT|MSI|MFW); MFW (NCSI 1.0.6)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce1:  mem 0xdc000000-0xddffffff irq 48 at device 0.1 on pci1
miibus1:  on bce1
brgphy1:  PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Using defaults for TSO: 65518/35/2048
bce1: Ethernet address: 00:26:b9:30:42:13
bce1:
<5>bce1: link state changed to DOWN
ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (4.6.4); Bufs (RX:2;TX:2;PG:; Flags (SPLT|MSI|MFW); MFW (NCSI 1.0.6)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib2:  at device 3.0 on pci0
pci2:  on pcib2
mpt0:  port 0xfc00-0xfcff mem 0xdf2ec000-0xdf2effff,0xdf2f0000-0xdf2fffff irq 32 at device 0.0 on pci2
mpt0: MPI Version=1.5.18.0
mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt0: 1 Active Volume (2 Max)
mpt0: 2 Hidden Drive Members (14 Max)
mpt0:vol0(mpt0:0:0): Settings ( Hot-Plug-Spares High-Priority-ReSync )
mpt0:vol0(mpt0:0:0): Using Spare Pool: 0
mpt0:vol0(mpt0:0:0): 2 Members:
pcib3:       (mpt0:1:9:0): Primary Online
      (mpt0:1:1:0): Secondary Online
mpt0:vol0(mpt0:0:0): RAID-1 - Optimal
mpt0:vol0(mpt0:0:0): Status ( Enabled )
(mpt0:vol0:0): Physical (mpt0:0:9:0), Pass-thru (mpt0:1:0:0)
(mpt0:vol0:0):  at device 7.0 on pci0
pci3: Online
(mpt0:vol0:1): Physical (mpt0:0:1:0), Pass-thru (mpt0:1:1:0)
(mpt0:vol0:1): Online
(noperiph:mpt0:1:-1:ffffffff): rescan already queued
 on pcib3
pci0:  at device 20.0 (no driver attached)
pci0:  at device 20.1 (no driver attached)
pci0:  at device 20.2 (no driver attached)
uhci0:  port 0xec40-0xec5f irq 17 at device 26.0 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1:  port 0xec60-0xec7f irq 18 at device 26.1 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
ehci0:  mem 0xdf0fe000-0xdf0fe3ff irq 19 at device 26.7 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci0
usbus2: 480Mbps High Speed USB v2.0
uhci2:  port 0xec80-0xec9f irq 21 at device 29.0 on pci0
usbus3 on uhci2
usbus3: 12Mbps Full Speed USB v1.0
uhci3:  port 0xeca0-0xecbf irq 20 at device 29.1 on pci0
usbus4 on uhci3
usbus4: 12Mbps Full Speed USB v1.0
uhci4:  port 0xecc0-0xecdf irq 21 at device 29.2 on pci0
usbus5 on uhci4
usbus5: 12Mbps Full Speed USB v1.0
uhci5:  port 0xece0-0xecff irq 20 at device 29.3 on pci0
usbus6 on uhci5
usbus6: 12Mbps Full Speed USB v1.0
ehci1:  mem 0xdf0ff000-0xdf0ff3ff irq 21 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
usbus7: 480Mbps High Speed USB v2.0
pcib4:  at device 30.0 on pci0
pci4:  on pcib4
vgapci0:  mem 0xd9800000-0xd9ffffff,0xde7fc000-0xde7fffff,0xde800000-0xdeffffff irq 19 at device 3.0 on pci4
vgapci0: Boot video device
isab0:  at device 31.0 on pci0
isa0:  on isab0
atapci0:  port 0xe8e0-0xe8e7,0xe8d0-0xe8d3,0xe8e8-0xe8ef,0xe8d4-0xe8d7,0xec00-0xec0f,0xec10-0xec1f irq 23 at device 31.2 on pci0
ata2:  at channel 0 on atapci0
ata3:  at channel 1 on atapci0
atapci1:  port 0xe8f0-0xe8f7,0xe8d8-0xe8db,0xe8f8-0xe8ff,0xe8dc-0xe8df,0xec20-0xec2f,0xec30-0xec3f irq 22 at device 31.5 on pci0
ata4:  at channel 0 on atapci1
ata5:  at channel 1 on atapci1
uart0:  port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (-1,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0:  at iomem 0xc0000-0xc7fff,0xec000-0xeffff on isa0
sc0:  at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0:  at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0:  at port 0x60,0x64 on isa0
atkbd0:  irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
est0:  on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 11
device_attach: est0 attach returned 6
est1:  on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 11
device_attach: est1 attach returned 6
est2:  on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 11
device_attach: est2 attach returned 6
est3:  on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 11
device_attach: est3 attach returned 6
Timecounters tick every 1.000 msec
(probe0:mpt0:0:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
ugen2.1:  at usbus2
ugen7.1:  at usbus7
ugen5.1:  at usbus5
ugen6.1:  at usbus6
uhub0:  on usbus5
uhub1:  on usbus7
ugen4.1:  at usbus4
ugen3.1:  at usbus3
uhub2:  on usbus6
(probe0:mpt0:0:0:0): CAM status: SCSI Status Error
uhub3: (probe0:mpt0:0:0:0): SCSI status: Check Condition
 on usbus4
uhub4:  on usbus2
ugen1.1:  at usbus1
ugen0.1:  at usbus0
(probe0:mpt0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:ffffffff,ffffffff (Reserved ASC/ASCQ pair)
uhub5:  on usbus3
uhub6:  on usbus1
uhub7: (probe0: on usbus0
mpt0:0:0:0): Error 22, Unretryable error
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Retrying command
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Retrying command
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Retrying command
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Retrying command
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Error 5, Retries exhausted
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub7: 2 ports with 2 removable, self powered
uhub1: 4 ports with 4 removable, self powered
uhub4: 4 ports with 4 removable, self powered
ses0 at mpt0 bus 0 scbus0 target 8 lun 0
da0 at mpt0 bus 0 scbus0 target 0 lun 0
ugen2.2:  at usbus2
ses0: da0:  Fixed Enclosure Services SPC-3 SCSI device
ses0: 300.000MB/s transfers
ses0: SCSI-3 ENC Device
uhub8 Fixed Direct Access SPC-3 SCSI device
 on uhub4
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 139392MB (285474816 512 byte sectors)
pass2 at mpt0 bus 1 scbus1 target 1 lun 0
uhub8: cd0 at ata4 bus 0 scbus4 target 0 lun 0
 on usbus2
pass2: cd0:  Removable CD-ROM SCSI device
cd0: Serial Number 908040307793
 Fixed Uninstalled SPC-3 SCSI device
cd0: 150.000MB/s transfersuhub8:  (MTT enabled
pass2: Serial Number 6SE3MMT7
pass2: 300.000MB/s transfers
SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
pass2: Command Queueing enabled
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
random: unblocking device.
uhub8: 4 ports with 4 removable, self powered
<118>Mounting filesystems...
<118>tunefs: soft updates remains unchanged as enabled
<118>tunefs: file system reloaded
<118>camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed
<118>** /dev/gpt/rootfs
<118>FILE SYSTEM CLEAN; SKIPPING CHECKS
<118>clean, 31941644 free (3252 frags, 3992299 blocks, 0.0% fragmentation)
<118>** /dev/gpt/rootfs
<118>FILE SYSTEM CLEAN; SKIPPING CHECKS
<118>clean, 31941644 free (3252 frags, 3992299 blocks, 0.0% fragmentation)
<118>Setting hostuuid: 4c4c4544-0039-3410-8058-b3c04f323253.
<118>Setting hostid: 0xda86f49c.
<118>Configuring syscons: blanktime.
<118>Setting up memory disks...done.
<118>Configuring crash dump device: /dev/gpt/swapfs
<118>swapon: adding /dev/gpt/swapfs as swap device
<118>.ELF ldconfig path: /lib /usr/lib /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.30/mach/CORE
<118>32-bit compatibility ldconfig path:
<118>done.
<118>>>> Invoking early script 'update'
<118>>>> Invoking early script 'configd'
<118>Starting configd.
<118>>>> Invoking early script 'templates'
<118>Generating configuration: OK
<118>>>> Invoking early script 'backup'
<118>>>> Invoking backup script 'captiveportal'
<118>>>> Invoking backup script 'dhcpleases'
<118>>>> Invoking backup script 'duid'
<118>>>> Invoking backup script 'netflow'
<118>>>> Invoking backup script 'rrd'
<118>>>> Invoking early script '98_carp_maintenance'
<118>>>> Invoking early script 'carp_service_status'
<118>OK
<118>Launching the init system...done.
<118>Initializing...........done.
<118>Starting device manager...done.
<118>Configuring login behaviour...done.
<118>Configuring loopback interface...done.
<118>Configuring kernel modules...done.
<118>Setting up extended sysctls...sysctl: oid 'kern.randompid' is read only
<118>sysctl: oid 'kern.randompid' is read only
<118>done.
<118>Setting timezone...done.
<118>Writing firmware setting...done.
<118>Writing trust files...done.
<118>Setting hostname: EXTOPS01.advext.local
<118>Generating /etc/hosts...done.
<118>Configuring system logging...done.
<118>Configuring loopback interface...done.
<118>Creating wireless clone interfaces...done.
<118>Configuring VLAN interfaces...
<6>vlan0: changing name to 'bce0_vlan50'
<6>vlan1: changing name to 'bce0_vlan10'
<6>vlan2: changing name to 'bce0_vlan1'
<118>done.
<118>Configuring LAN interface...
<6>bce0: promiscuous mode enabled
<6>bce0_vlan10: promiscuous mode enabled
<6>carp: demoted by 240 to 240 (interface down)
<6>carp: demoted by 240 to 480 (pfsync bulk start)
<118>done.
<118>Configuring SYNC interface...done.
<118>Configuring WAN interface...
<6>bce0_vlan50: promiscuous mode enabled
<6>carp: demoted by 240 to 720 (interface down)
<6>carp: demoted by 240 to 960 (interface down)
<6>carp: demoted by 240 to 1200 (interface down)
<6>carp: demoted by 240 to 1440 (interface down)
<6>carp: demoted by 240 to 1680 (interface down)
<6>carp: demoted by 240 to 1920 (interface down)
<6>carp: demoted by 240 to 2160 (interface down)
<6>carp: demoted by 240 to 2400 (interface down)
<6>carp: demoted by 240 to 2640 (interface down)
<6>carp: demoted by 240 to 2880 (interface down)
<6>carp: demoted by 240 to 3120 (interface down)
<6>carp: demoted by 240 to 3360 (interface down)
<118>done.
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan10: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan10: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan10: 3
<118>Configuring CARP settings...done.
<118>Creating IPsec VTI instances...done.
<118>Generating /etc/resolv.conf...done.
<118>Configuring firewall.......done.
<118>Starting PFLOG...done.
<6>pflog0: promiscuous mode enabled
<118>Configuring OpenSSH...done.
<118>Starting web GUI...done.
<118>Configuring CRON...done.
<118>Setting up routes...done.
<118>Generating /etc/hosts...done.
<118>Starting Dnsmasq DNS...done.
<118>Setting up gateway monitors...done.
<118>Configuring firewall.......done.
<118>Starting PFLOG...
bce0:
<5>bce0: link state changed to UP
<5>bce0_vlan1: link state changed to UP
<6>carp: 4@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 3120 (interface up)
<6>carp: 5@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 2880 (interface up)
<6>carp: 14@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 2640 (interface up)
<6>carp: 6@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 2400 (interface up)
<6>carp: 7@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 2160 (interface up)
<6>carp: 13@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 1920 (interface up)
<6>carp: 8@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 1680 (interface up)
<6>carp: 9@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 1440 (interface up)
<6>carp: 10@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 1200 (interface up)
<6>carp: 11@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 960 (interface up)
<6>carp: 12@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 720 (interface up)
<6>carp: 3@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 480 (interface up)
<5>bce0_vlan50: link state changed to UP
<6>carp: 2@bce0_vlan10: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 240 (interface up)
<5>bce0_vlan10: link state changed to UP
Gigabit link up!
<6>pflog0: promiscuous mode disabled
bce0: Gigabit link up!
<118>done.
<6>pflog0: promiscuous mode enabled
bce1:
<5>bce1: link state changed to UP
Gigabit link up!
bce1: Gigabit link up!
<118>Starting NTP service...deferred.
<118>Generating RRD graphs...done.
<118>Configuring system logging...
<6>pflog0: promiscuous mode disabled
<6>pflog0: promiscuous mode enabled
<6>carp: demoted by -240 to 0 (pfsync bulk done)
<118>done.
<118>>>> Invoking start script 'newwanip'
<118>>>> Invoking start script 'freebsd'
<118>osrelayd is not running.
<118>Starting osrelayd.
<118>Cannot get pid for /usr/local/sbin/relayd
<118>>>> Invoking start script 'carp'
<118>>>> Invoking start script 'cron'
<118>Starting Cron: OK
<118>>>> Invoking start script 'beep'
<6>carp: 7@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 4@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 5@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 14@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 3@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 2@bce0_vlan10: BACKUP -> MASTER (preempting a slower master)
<6>carp: 13@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 8@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 9@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 10@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 11@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 12@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 6@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<118>Root file system: /dev/gpt/rootfs
<118>Sun Jan  5 14:31:46 +08 2020
<118>
<118>*** EXTOPS01.advext.local: OPNsense 19.7.8 (amd64/OpenSSL) ***
<118>
<118> LAN (bce0_vlan10) -> v4: 172.25.15.201/20
<118> SYNC (bce1)     -> v4: 192.168.0.3/24
<118> WAN (bce0_vlan50) -> v4: 172.25.50.150/24
<118>
<118> SSH:   SHA256 kogUQi3OQmX2jwpgjAiGrIYSXC0xVD0bO1Y+/mD768A (ECDSA)
<118> SSH:   SHA256 AHYn78p6ZUnY4n5nwTbYMID+XYrW5qbbFqvmn/jWceo (ED25519)
<118> SSH:   SHA256 v070cNQsRvQjRq6FF0Q59yDSkD+p0KTzMDDPnueeqX4 (RSA)
<6>pflog0: promiscuous mode disabled
<6>pflog0: promiscuous mode enabled
<6>pid 46141 (python3.7), uid 0: exited on signal 11 (core dumped)
[HBSD SEGVGUARD] [/usr/local/bin/python3 (95280)] Suspension expired.
 -> pid: 95280 ppid: 91060 p_pax: 0xa50


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 14
fault virtual address   = 0xfffffffe80a120f7
fault code      = supervisor read data, page not present
instruction pointer   = 0x20:0xfffffffe80a120f7
stack pointer           = 0x28:0xfffffe0237f39b80
frame pointer           = 0x28:0xfffffe0237f39bb0
code segment      = base 0x0, limit 0xfffff, type 0x1b
         = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process      = 15 (usbus7)
version.txt0600007013604326477  7544 ustarrootwheelFreeBSD 11.2-RELEASE-p16-HBSD  87a7fc985c3(stable/19.7)Edit (fabian): Wrap log in code tags to allow scoling / folding.

[fabian]

If you look at the error message:

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 14
fault virtual address   = 0xfffffffe80a120f7
fault code      = supervisor read data, page not present
...
current process      = 15 (usbus7)

For me, there are two possible issues:

• You have a memory problem
• You have a faulty driver or device (usbus device -> don't know freebsd naming convention but do you have some USB deviced attached as it sounds a bit like that?)

[javier.wong]

Thanks for the suggestion @fabian. The server currently does not have any USB devices attached to it. Also I have replaced the RAM DIMMS but to no avail.

Would it be anything else or are you absolutely certain it's one of the 2 things you've mentioned? There is definitely no USB device so it has to be some faulty memory bank(s) if that is the case.

[fabian]

No drivers as well.

[Darkopnsense]

hi javier,

My question, what is the value of the disk swap, please?

Regards,

[javier.wong]

Swap is at 8GB. Utilization of swap is 0.

Thanks for the suggestion guys.

[Darkopnsense]

Hello javier,

Your values don't confirm what I thought.

Do you know the health of your hard drives?

The Status Smart plug-in will allow you to take stock.

Cordially.