Hi guys,
I have a clustered pair of FWs. Some months back the primary firewall started to crash frequently almost on a daily basis. The secondary FW does not crash at all. The primary firewall is on a Dell server whereas the secondary one is on a low end network appliance. I have tried the following to no avail:
- Reformatted the FW and reloaded the configuration
- Changed the RAM DIMMS
- Upgraded to the latest version 19.7.8
Does anyone have any idea what may be causing this?
Thanks in advance.
Here is the crash log (had to remove the top portion due to the message limit):
Tracing command devd pid 99454 tid 100121 td 0xfffff80006269000
sched_switch() at sched_switch+0x8ad/frame 0xfffffe02380d9720
mi_switch() at mi_switch+0xe6/frame 0xfffffe02380d9750
sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe02380d97b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe02380d97f0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x18c/frame 0xfffffe02380d9850
seltdwait() at seltdwait+0x7b/frame 0xfffffe02380d9890
kern_select() at kern_select+0x850/frame 0xfffffe02380d9a80
sys_select() at sys_select+0x56/frame 0xfffffe02380d9ac0
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe02380d9bf0
fast_syscall_common() at fast_syscall_common+0x101/frconfig.txt0600001103513604326477 7547 ustarrootwheeloptions CONFIG_AUTOGENERATED
ident SMP
machine amd64
cpu HAMMER
makeoptions WITH_CTF=1
options PAX_JAIL_SUPPORT
options PAX_CONTROL_EXTATTR
options PAX_CONTROL_ACL_OVERRIDE_SUPPORT
options PAX_CONTROL_ACL
options PAX_INSECURE_MODE
options PAX_SEGVGUARD
options PAX_HARDENING
options PAX_ASLR
options PAX
options TCP_SIGNATURE
options PPS_SYNC
options MROUTING
options IPSTEALTH
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
options GEOM_UZIP
options GEOM_MIRROR
options GEOM_ELI
options GEOM_BDE
options DDB
options XENHVM
options USB_DEBUG
options ATH_ENABLE_11N
options AH_AR5416_INTERRUPT_MITIGATION
options AH_SUPPORT_AR5416
options IEEE80211_SUPPORT_MESH
options IEEE80211_AMPDU_AGE
options IEEE80211_DEBUG
options SC_PIXEL_MODE
options VESA
options AHD_REG_PRETTY_PRINT
options AHC_REG_PRETTY_PRINT
options PCI_IOV
options PCI_HP
options ACPI_DMAR
options EARLY_AP_STARTUP
options DEVICE_NUMA
options SMP
options KDB_TRACE
options KDB
options RCTL
options RACCT_DEFAULT_TO_DISABLED
options RACCT
options INCLUDE_CONFIG_FILE
options DDB_CTF
options KDTRACE_HOOKS
options KDTRACE_FRAME
options MAC
options CAPABILITIES
options CAPABILITY_MODE
options AUDIT
options HWPMC_HOOKS
options KBD_INSTALL_CDEV
options PRINTF_BUFR_SIZE=128
options _KPOSIX_PRIORITY_SCHEDULING
options SYSVSEM
options SYSVMSG
options SYSVSHM
options STACK
options KTRACE
options SCSI_DELAY=5000
options COMPAT_FREEBSD10
options COMPAT_FREEBSD9
options COMPAT_FREEBSD7
options COMPAT_FREEBSD6
options COMPAT_FREEBSD5
options COMPAT_FREEBSD4
options COMPAT_FREEBSD32
options GEOM_LABEL
options GEOM_RAID
options GEOM_PART_GPT
options PSEUDOFS
options PROCFS
options CD9660
options MSDOSFS
options NFS_ROOT
options NFSLOCKD
options NFSD
options NFSCL
options MD_ROOT
options QUOTA
options UFS_GJOURNAL
options UFS_DIRHASH
options UFS_ACL
options SOFTUPDATES
options FFS
options SCTP
options TCP_OFFLOAD
options IPSEC_SUPPORT
options IPSEC
options INET6
options INET
options PREEMPTION
options SCHED_ULE
options NEW_PCIB
options GEOM_PART_MBR
options GEOM_PART_EBR_COMPAT
options GEOM_PART_EBR
options GEOM_PART_BSD
device isa
device mem
device io
device uart_ns8250
device cpufreq
device acpi
device pci
device fdc
device ahci
device ata
device mvs
device siis
device ahc
device ahd
device esp
device hptiop
device isp
device mpt
device mps
device mpr
device sym
device trm
device adv
device adw
device aic
device bt
device isci
device ocs_fc
device scbus
device ch
device da
device sa
device cd
device pass
device ses
device amr
device arcmsr
device ciss
device dpt
device hptmv
device hptnr
device hptrr
device hpt27xx
device iir
device ips
device mly
device twa
device smartpqi
device tws
device aac
device aacp
device aacraid
device ida
device mfi
device mlx
device mrsas
device pmspcv
device twe
device nvme
device nvd
device atkbdc
device atkbd
device psm
device kbdmux
device vga
device splash
device sc
device vt
device vt_vga
device vt_efifb
device agp
device cbb
device pccard
device cardbus
device uart
device ppc
device ppbus
device lpt
device ppi
device puc
device bxe
device de
device em
device igb
device ix
device ixv
device ixl
device ixlv
device le
device ti
device txp
device vx
device miibus
device ae
device age
device alc
device ale
device bce
device bfe
device bge
device cas
device dc
device et
device fxp
device gem
device hme
device jme
device lge
device msk
device nfe
device nge
device pcn
device re
device rl
device sf
device sge
device sis
device sk
device ste
device stge
device tl
device tx
device vge
device vr
device wb
device xl
device wlan
device wlan_wep
device wlan_ccmp
device wlan_tkip
device wlan_amrr
device an
device ath
device ath_pci
device ath_hal
device ath_rate_sample
device ipw
device iwi
device iwn
device malo
device mwl
device ral
device wi
device wpi
device loop
device random
device padlock_rng
device rdrand_rng
device ether
device vlan
device tun
device md
device gif
device firmware
device bpf
device uhci
device ohci
device ehci
device xhci
device usb
device ukbd
device umass
device sound
device snd_cmi
device snd_csa
device snd_emu10kx
device snd_es137x
device snd_hda
device snd_ich
device snd_via8233
device mmc
device mmcsd
device sdhci
device virtio
device virtio_pci
device vtnet
device virtio_blk
device virtio_scsi
device virtio_balloon
device hyperv
device xenpci
device vmx
device netmap
device crypto
device bwi
device bwn
device rum
device run
device siba_bwn
device u3g
device uark
device uath
device uftdi
device umct
device umodem
device upgt
device uplcom
device ural
device urtw
device uvisor
device uvscom
device zyd
device wlan_acl
device wlan_xauth
device speaker
msgbuf.txt0600005645713604326477 7606 ustarrootwheelCopyright (c) 2013-2018 The HardenedBSD Project.
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-RELEASE-p16-HBSD 87a7fc985c3(stable/19.7) amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
HardenedBSD: initialize and check features (__HardenedBSD_version 1100056 __FreeBSD_version 1102000).
CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2261.05-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x106a5 Family=0x6 Model=0x1a Stepping=5
Features=0xbfebfbff
Features2=0x9ce3bd
AMD Features=0x28000800
AMD Features2=0x1
VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,VPID
TSC: P-state invariant, performance statistics
real memory = 8589934592 (8192 MB)
avail memory = 8244363264 (7862 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table:
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
ioapic1: Changing APIC ID to 1
ioapic0 irqs 0-23 on motherboard
ioapic1 irqs 32-55 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1130522594 Hz quality 1000
random: entropy device external interface
wlan: mac acl policy registered
000.000017 [4213] netmap_init netmap: loaded module
kbd1 at kbdmux0
nexus0
cryptosoft0: on motherboard
acpi0: on motherboard
acpi0: Power Button (fixed)
cpu0: on acpi0
cpu1: on acpi0
cpu2: on acpi0
cpu3: on acpi0
atrtc0: port 0x70-0x7f irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: port 0x40-0x5f irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: port 0xcf8-0xcff on acpi0
pcib0: _OSC returned error 0x10
pci0: on pcib0
pcib1: at device 1.0 on pci0
pci1: on pcib1
bce0: mem 0xda000000-0xdbffffff irq 36 at device 0.0 on pci1
miibus0: on bce0
brgphy0: PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce0: Using defaults for TSO: 65518/35/2048
bce0: Ethernet address: 00:26:b9:30:42:12
bce0:
<5>bce0: link state changed to DOWN
ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (4.6.4); Bufs (RX:2;TX:2;PG:; Flags (SPLT|MSI|MFW); MFW (NCSI 1.0.6)
Coal (RX:6,6,18,18; TX:20,20,80,80)
bce1: mem 0xdc000000-0xddffffff irq 48 at device 0.1 on pci1
miibus1: on bce1
brgphy1: PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce1: Using defaults for TSO: 65518/35/2048
bce1: Ethernet address: 00:26:b9:30:42:13
bce1:
<5>bce1: link state changed to DOWN
ASIC (0x57092008); Rev (C0); Bus (PCIe x4, 2.5Gbps); B/C (4.6.4); Bufs (RX:2;TX:2;PG:; Flags (SPLT|MSI|MFW); MFW (NCSI 1.0.6)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib2: at device 3.0 on pci0
pci2: on pcib2
mpt0: port 0xfc00-0xfcff mem 0xdf2ec000-0xdf2effff,0xdf2f0000-0xdf2fffff irq 32 at device 0.0 on pci2
mpt0: MPI Version=1.5.18.0
mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt0: 1 Active Volume (2 Max)
mpt0: 2 Hidden Drive Members (14 Max)
mpt0:vol0(mpt0:0:0): Settings ( Hot-Plug-Spares High-Priority-ReSync )
mpt0:vol0(mpt0:0:0): Using Spare Pool: 0
mpt0:vol0(mpt0:0:0): 2 Members:
pcib3: (mpt0:1:9:0): Primary Online
(mpt0:1:1:0): Secondary Online
mpt0:vol0(mpt0:0:0): RAID-1 - Optimal
mpt0:vol0(mpt0:0:0): Status ( Enabled )
(mpt0:vol0:0): Physical (mpt0:0:9:0), Pass-thru (mpt0:1:0:0)
(mpt0:vol0:0): at device 7.0 on pci0
pci3: Online
(mpt0:vol0:1): Physical (mpt0:0:1:0), Pass-thru (mpt0:1:1:0)
(mpt0:vol0:1): Online
(noperiph:mpt0:1:-1:ffffffff): rescan already queued
on pcib3
pci0: at device 20.0 (no driver attached)
pci0: at device 20.1 (no driver attached)
pci0: at device 20.2 (no driver attached)
uhci0: port 0xec40-0xec5f irq 17 at device 26.0 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1: port 0xec60-0xec7f irq 18 at device 26.1 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
ehci0: mem 0xdf0fe000-0xdf0fe3ff irq 19 at device 26.7 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci0
usbus2: 480Mbps High Speed USB v2.0
uhci2: port 0xec80-0xec9f irq 21 at device 29.0 on pci0
usbus3 on uhci2
usbus3: 12Mbps Full Speed USB v1.0
uhci3: port 0xeca0-0xecbf irq 20 at device 29.1 on pci0
usbus4 on uhci3
usbus4: 12Mbps Full Speed USB v1.0
uhci4: port 0xecc0-0xecdf irq 21 at device 29.2 on pci0
usbus5 on uhci4
usbus5: 12Mbps Full Speed USB v1.0
uhci5: port 0xece0-0xecff irq 20 at device 29.3 on pci0
usbus6 on uhci5
usbus6: 12Mbps Full Speed USB v1.0
ehci1: mem 0xdf0ff000-0xdf0ff3ff irq 21 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
usbus7: 480Mbps High Speed USB v2.0
pcib4: at device 30.0 on pci0
pci4: on pcib4
vgapci0: mem 0xd9800000-0xd9ffffff,0xde7fc000-0xde7fffff,0xde800000-0xdeffffff irq 19 at device 3.0 on pci4
vgapci0: Boot video device
isab0: at device 31.0 on pci0
isa0: on isab0
atapci0: port 0xe8e0-0xe8e7,0xe8d0-0xe8d3,0xe8e8-0xe8ef,0xe8d4-0xe8d7,0xec00-0xec0f,0xec10-0xec1f irq 23 at device 31.2 on pci0
ata2: at channel 0 on atapci0
ata3: at channel 1 on atapci0
atapci1: port 0xe8f0-0xe8f7,0xe8d8-0xe8db,0xe8f8-0xe8ff,0xe8dc-0xe8df,0xec20-0xec2f,0xec30-0xec3f irq 22 at device 31.5 on pci0
ata4: at channel 0 on atapci1
ata5: at channel 1 on atapci1
uart0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (-1,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: at iomem 0xc0000-0xc7fff,0xec000-0xeffff on isa0
sc0: at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: at port 0x60,0x64 on isa0
atkbd0: irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
est0: on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 11
device_attach: est0 attach returned 6
est1: on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 11
device_attach: est1 attach returned 6
est2: on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 11
device_attach: est2 attach returned 6
est3: on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 11
device_attach: est3 attach returned 6
Timecounters tick every 1.000 msec
(probe0:mpt0:0:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
ugen2.1: at usbus2
ugen7.1: at usbus7
ugen5.1: at usbus5
ugen6.1: at usbus6
uhub0: on usbus5
uhub1: on usbus7
ugen4.1: at usbus4
ugen3.1: at usbus3
uhub2: on usbus6
(probe0:mpt0:0:0:0): CAM status: SCSI Status Error
uhub3: (probe0:mpt0:0:0:0): SCSI status: Check Condition
on usbus4
uhub4: on usbus2
ugen1.1: at usbus1
ugen0.1: at usbus0
(probe0:mpt0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:ffffffff,ffffffff (Reserved ASC/ASCQ pair)
uhub5: on usbus3
uhub6: on usbus1
uhub7: (probe0: on usbus0
mpt0:0:0:0): Error 22, Unretryable error
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Retrying command
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Retrying command
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Retrying command
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Retrying command
(probe13:mpt0:1:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe13:mpt0:1:0:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe13:mpt0:1:0:0): Error 5, Retries exhausted
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub7: 2 ports with 2 removable, self powered
uhub1: 4 ports with 4 removable, self powered
uhub4: 4 ports with 4 removable, self powered
ses0 at mpt0 bus 0 scbus0 target 8 lun 0
da0 at mpt0 bus 0 scbus0 target 0 lun 0
ugen2.2: at usbus2
ses0: da0: Fixed Enclosure Services SPC-3 SCSI device
ses0: 300.000MB/s transfers
ses0: SCSI-3 ENC Device
uhub8 Fixed Direct Access SPC-3 SCSI device
on uhub4
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 139392MB (285474816 512 byte sectors)
pass2 at mpt0 bus 1 scbus1 target 1 lun 0
uhub8: cd0 at ata4 bus 0 scbus4 target 0 lun 0
on usbus2
pass2: cd0: Removable CD-ROM SCSI device
cd0: Serial Number 908040307793
Fixed Uninstalled SPC-3 SCSI device
cd0: 150.000MB/s transfersuhub8: (MTT enabled
pass2: Serial Number 6SE3MMT7
pass2: 300.000MB/s transfers
SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
pass2: Command Queueing enabled
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
random: unblocking device.
uhub8: 4 ports with 4 removable, self powered
<118>Mounting filesystems...
<118>tunefs: soft updates remains unchanged as enabled
<118>tunefs: file system reloaded
<118>camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed
<118>** /dev/gpt/rootfs
<118>FILE SYSTEM CLEAN; SKIPPING CHECKS
<118>clean, 31941644 free (3252 frags, 3992299 blocks, 0.0% fragmentation)
<118>** /dev/gpt/rootfs
<118>FILE SYSTEM CLEAN; SKIPPING CHECKS
<118>clean, 31941644 free (3252 frags, 3992299 blocks, 0.0% fragmentation)
<118>Setting hostuuid: 4c4c4544-0039-3410-8058-b3c04f323253.
<118>Setting hostid: 0xda86f49c.
<118>Configuring syscons: blanktime.
<118>Setting up memory disks...done.
<118>Configuring crash dump device: /dev/gpt/swapfs
<118>swapon: adding /dev/gpt/swapfs as swap device
<118>.ELF ldconfig path: /lib /usr/lib /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.30/mach/CORE
<118>32-bit compatibility ldconfig path:
<118>done.
<118>>>> Invoking early script 'update'
<118>>>> Invoking early script 'configd'
<118>Starting configd.
<118>>>> Invoking early script 'templates'
<118>Generating configuration: OK
<118>>>> Invoking early script 'backup'
<118>>>> Invoking backup script 'captiveportal'
<118>>>> Invoking backup script 'dhcpleases'
<118>>>> Invoking backup script 'duid'
<118>>>> Invoking backup script 'netflow'
<118>>>> Invoking backup script 'rrd'
<118>>>> Invoking early script '98_carp_maintenance'
<118>>>> Invoking early script 'carp_service_status'
<118>OK
<118>Launching the init system...done.
<118>Initializing...........done.
<118>Starting device manager...done.
<118>Configuring login behaviour...done.
<118>Configuring loopback interface...done.
<118>Configuring kernel modules...done.
<118>Setting up extended sysctls...sysctl: oid 'kern.randompid' is read only
<118>sysctl: oid 'kern.randompid' is read only
<118>done.
<118>Setting timezone...done.
<118>Writing firmware setting...done.
<118>Writing trust files...done.
<118>Setting hostname: EXTOPS01.advext.local
<118>Generating /etc/hosts...done.
<118>Configuring system logging...done.
<118>Configuring loopback interface...done.
<118>Creating wireless clone interfaces...done.
<118>Configuring VLAN interfaces...
<6>vlan0: changing name to 'bce0_vlan50'
<6>vlan1: changing name to 'bce0_vlan10'
<6>vlan2: changing name to 'bce0_vlan1'
<118>done.
<118>Configuring LAN interface...
<6>bce0: promiscuous mode enabled
<6>bce0_vlan10: promiscuous mode enabled
<6>carp: demoted by 240 to 240 (interface down)
<6>carp: demoted by 240 to 480 (pfsync bulk start)
<118>done.
<118>Configuring SYNC interface...done.
<118>Configuring WAN interface...
<6>bce0_vlan50: promiscuous mode enabled
<6>carp: demoted by 240 to 720 (interface down)
<6>carp: demoted by 240 to 960 (interface down)
<6>carp: demoted by 240 to 1200 (interface down)
<6>carp: demoted by 240 to 1440 (interface down)
<6>carp: demoted by 240 to 1680 (interface down)
<6>carp: demoted by 240 to 1920 (interface down)
<6>carp: demoted by 240 to 2160 (interface down)
<6>carp: demoted by 240 to 2400 (interface down)
<6>carp: demoted by 240 to 2640 (interface down)
<6>carp: demoted by 240 to 2880 (interface down)
<6>carp: demoted by 240 to 3120 (interface down)
<6>carp: demoted by 240 to 3360 (interface down)
<118>done.
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan50: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan10: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan10: 3
<7>ifa_maintain_loopback_route: deletion failed for interface bce0_vlan10: 3
<118>Configuring CARP settings...done.
<118>Creating IPsec VTI instances...done.
<118>Generating /etc/resolv.conf...done.
<118>Configuring firewall.......done.
<118>Starting PFLOG...done.
<6>pflog0: promiscuous mode enabled
<118>Configuring OpenSSH...done.
<118>Starting web GUI...done.
<118>Configuring CRON...done.
<118>Setting up routes...done.
<118>Generating /etc/hosts...done.
<118>Starting Dnsmasq DNS...done.
<118>Setting up gateway monitors...done.
<118>Configuring firewall.......done.
<118>Starting PFLOG...
bce0:
<5>bce0: link state changed to UP
<5>bce0_vlan1: link state changed to UP
<6>carp: 4@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 3120 (interface up)
<6>carp: 5@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 2880 (interface up)
<6>carp: 14@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 2640 (interface up)
<6>carp: 6@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 2400 (interface up)
<6>carp: 7@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 2160 (interface up)
<6>carp: 13@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 1920 (interface up)
<6>carp: 8@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 1680 (interface up)
<6>carp: 9@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 1440 (interface up)
<6>carp: 10@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 1200 (interface up)
<6>carp: 11@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 960 (interface up)
<6>carp: 12@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 720 (interface up)
<6>carp: 3@bce0_vlan50: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 480 (interface up)
<5>bce0_vlan50: link state changed to UP
<6>carp: 2@bce0_vlan10: INIT -> BACKUP (initialization complete)
<6>carp: demoted by -240 to 240 (interface up)
<5>bce0_vlan10: link state changed to UP
Gigabit link up!
<6>pflog0: promiscuous mode disabled
bce0: Gigabit link up!
<118>done.
<6>pflog0: promiscuous mode enabled
bce1:
<5>bce1: link state changed to UP
Gigabit link up!
bce1: Gigabit link up!
<118>Starting NTP service...deferred.
<118>Generating RRD graphs...done.
<118>Configuring system logging...
<6>pflog0: promiscuous mode disabled
<6>pflog0: promiscuous mode enabled
<6>carp: demoted by -240 to 0 (pfsync bulk done)
<118>done.
<118>>>> Invoking start script 'newwanip'
<118>>>> Invoking start script 'freebsd'
<118>osrelayd is not running.
<118>Starting osrelayd.
<118>Cannot get pid for /usr/local/sbin/relayd
<118>>>> Invoking start script 'carp'
<118>>>> Invoking start script 'cron'
<118>Starting Cron: OK
<118>>>> Invoking start script 'beep'
<6>carp: 7@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 4@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 5@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 14@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 3@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 2@bce0_vlan10: BACKUP -> MASTER (preempting a slower master)
<6>carp: 13@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 8@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 9@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 10@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 11@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 12@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<6>carp: 6@bce0_vlan50: BACKUP -> MASTER (preempting a slower master)
<118>Root file system: /dev/gpt/rootfs
<118>Sun Jan 5 14:31:46 +08 2020
<118>
<118>*** EXTOPS01.advext.local: OPNsense 19.7.8 (amd64/OpenSSL) ***
<118>
<118> LAN (bce0_vlan10) -> v4: 172.25.15.201/20
<118> SYNC (bce1) -> v4: 192.168.0.3/24
<118> WAN (bce0_vlan50) -> v4: 172.25.50.150/24
<118>
<118> SSH: SHA256 kogUQi3OQmX2jwpgjAiGrIYSXC0xVD0bO1Y+/mD768A (ECDSA)
<118> SSH: SHA256 AHYn78p6ZUnY4n5nwTbYMID+XYrW5qbbFqvmn/jWceo (ED25519)
<118> SSH: SHA256 v070cNQsRvQjRq6FF0Q59yDSkD+p0KTzMDDPnueeqX4 (RSA)
<6>pflog0: promiscuous mode disabled
<6>pflog0: promiscuous mode enabled
<6>pid 46141 (python3.7), uid 0: exited on signal 11 (core dumped)
[HBSD SEGVGUARD] [/usr/local/bin/python3 (95280)] Suspension expired.
-> pid: 95280 ppid: 91060 p_pax: 0xa50
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 14
fault virtual address = 0xfffffffe80a120f7
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xfffffffe80a120f7
stack pointer = 0x28:0xfffffe0237f39b80
frame pointer = 0x28:0xfffffe0237f39bb0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 15 (usbus7)
version.txt0600007013604326477 7544 ustarrootwheelFreeBSD 11.2-RELEASE-p16-HBSD 87a7fc985c3(stable/19.7)
Edit (fabian): Wrap log in code tags to allow scoling / folding.
If you look at the error message:
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 14
fault virtual address = 0xfffffffe80a120f7
fault code = supervisor read data, page not present
...
current process = 15 (usbus7)
For me, there are two possible issues:
- You have a memory problem
- You have a faulty driver or device (usbus device -> don't know freebsd naming convention but do you have some USB deviced attached as it sounds a bit like that?)
Thanks for the suggestion @fabian. The server currently does not have any USB devices attached to it. Also I have replaced the RAM DIMMS but to no avail.
Would it be anything else or are you absolutely certain it's one of the 2 things you've mentioned? There is definitely no USB device so it has to be some faulty memory bank(s) if that is the case.
No drivers as well.
hi javier,
My question, what is the value of the disk swap, please?
Regards,
Swap is at 8GB. Utilization of swap is 0.
Thanks for the suggestion guys.
Hello javier,
Your values don't confirm what I thought.
Do you know the health of your hard drives?
The Status Smart plug-in will allow you to take stock.
Cordially.