Sudden kernel trap after domain_add(netgraph) after domainfinalize()

[DKSF]

Hi - I'm running into kernel traps that I need some help with diagnosing. IPv6 is turned off on all interfaces. Any suggestions on where to look? This is a newly installed clean updated 22.1.10.

The trigger seems to be this:

Code Select Expand

WARNING: attempt to domain_add(netgraph) after domainfinalize()
kernel trap 1 with interrupts disabled

Any pointers?

The full report from dmesg is as follows.

Code Select Expand

FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(efifb): resolution 1024x768
CPU: Intel(R) Celeron(R) N5105 @ 2.00GHz (1996.83-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906c0  Family=0x6  Model=0x9c  Stepping=0
  Features=0x1f83fbff
  Features2=0xcff8a223
  AMD Features=0x28100800
  AMD Features2=0x101
  Structured Extended Features=0x21940283
  Structured Extended Features2=0x18400124
  Structured Extended Features3=0xac000400
  XSAVE Features=0xf
  IA32_ARCH_CAPS=0x6b
  AMD Extended Feature Extensions ID EBX=0x100d000
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Hypervisor: Origin = "KVMKVMKVM"
real memory  = 8589934592 (8192 MB)
avail memory = 8235425792 (7853 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table:
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0  irqs 0-23
Launching APs: 1
random: entropy device external interface
wlan: mac acl policy registered
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
kvmclock0:
Timecounter "kvmclock" frequency 1000000000 Hz quality 975
kvmclock0: registered as a time-of-day clock, resolution 0.000001s
efirtc0:
efirtc0: registered as a time-of-day clock, resolution 1.000000s
aesni0:
acpi0:
acpi0: Power Button (fixed)
cpu0:  numa-domain 0 on acpi0
atrtc0:  port 0x70-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
vgapci0:  mem 0xc0000000-0xc0ffffff,0xc224b000-0xc224bfff at device 1.0 on pci0
vgapci0: Boot video device
uhci0:  port 0xd300-0xd31f irq 16 at device 26.0 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1:  port 0xd2e0-0xd2ff irq 17 at device 26.1 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2:  port 0xd2c0-0xd2df irq 18 at device 26.2 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
ehci0:  mem 0xc224a000-0xc224afff irq 19 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
hdac0:  mem 0xc2240000-0xc2243fff irq 16 at device 27.0 on pci0
pcib1:  mem 0xc2249000-0xc2249fff irq 16 at device 28.0 on pci0
pcib1: failed to allocate initial I/O port window: 0xd000-0xdfff
pci1:  on pcib1
pcib2:  mem 0xc2248000-0xc2248fff irq 16 at device 28.1 on pci0
pci2:  on pcib2
pcib3:  mem 0xc2247000-0xc2247fff irq 16 at device 28.2 on pci0
pci3:  on pcib3
pcib4:  mem 0xc2246000-0xc2246fff irq 16 at device 28.3 on pci0
pci4:  on pcib4
uhci3:  port 0xd2a0-0xd2bf irq 16 at device 29.0 on pci0
usbus4 on uhci3
usbus4: 12Mbps Full Speed USB v1.0
uhci4:  port 0xd280-0xd29f irq 17 at device 29.1 on pci0
usbus5 on uhci4
usbus5: 12Mbps Full Speed USB v1.0
uhci5:  port 0xd260-0xd27f irq 18 at device 29.2 on pci0
usbus6 on uhci5
usbus6: 12Mbps Full Speed USB v1.0
ehci1:  mem 0xc2245000-0xc2245fff irq 19 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
usbus7: 480Mbps High Speed USB v2.0
pcib5:  at device 30.0 on pci0
pci5:  on pcib5
pcib6:  mem 0xc1800000-0xc18000ff irq 21 at device 1.0 on pci5
pci6:  on pcib6
virtio_pci0:  port 0x9080-0x90bf mem 0x800014000-0x800017fff irq 20 at device 3.0 on pci6
vtballoon0:  on virtio_pci0
virtio_pci1:  port 0x9040-0x907f mem 0xc1604000-0xc1604fff,0x800010000-0x800013fff irq 22 at device 5.0 on pci6
vtscsi0:  on virtio_pci1
virtio_pci2:  port 0x9000-0x903f mem 0xc1603000-0xc1603fff,0x80000c000-0x80000ffff irq 21 at device 8.0 on pci6
virtio_pci3:  port 0x9100-0x911f mem 0xc1602000-0xc1602fff,0x800008000-0x80000bfff irq 23 at device 18.0 on pci6
vtnet0:  on virtio_pci3
vtnet0: Ethernet address: 1a:a9:c8:38:17:22
vtnet0: netmap queues/slots: TX 1/256, RX 1/128
000.000121 [ 450] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=128
virtio_pci4:  port 0x90e0-0x90ff mem 0xc1601000-0xc1601fff,0x800004000-0x800007fff irq 20 at device 19.0 on pci6
vtnet1:  on virtio_pci4
vtnet1: Ethernet address: 00:e0:4c:92:86:72
vtnet1: netmap queues/slots: TX 1/256, RX 1/128
000.000122 [ 450] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=128
virtio_pci5:  port 0x90c0-0x90df mem 0xc1600000-0xc1600fff,0x800000000-0x800003fff irq 21 at device 20.0 on pci6
vtnet2:  on virtio_pci5
vtnet2: Ethernet address: ce:7a:38:20:57:21
vtnet2: netmap queues/slots: TX 1/256, RX 1/128
000.000123 [ 450] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=128
pcib7:  mem 0xc1801000-0xc18010ff irq 22 at device 2.0 on pci5
pci7:  on pcib7
pcib8:  mem 0xc1802000-0xc18020ff irq 23 at device 3.0 on pci5
pci8:  on pcib8
pcib9:  mem 0xc1803000-0xc18030ff irq 20 at device 4.0 on pci5
pci9:  on pcib9
isab0:  at device 31.0 on pci0
isa0:  on isab0
ahci0:  port 0xd240-0xd25f mem 0xc2244000-0xc2244fff irq 16 at device 31.2 on pci0
ahci0: AHCI v1.00 with 6 1.5Gbps ports, Port Multiplier not supported
ahcich0:  at channel 0 on ahci0
ahcich1:  at channel 1 on ahci0
ahcich2:  at channel 2 on ahci0
ahcich3:  at channel 3 on ahci0
ahcich4:  at channel 4 on ahci0
ahcich5:  at channel 5 on ahci0
acpi_syscontainer0:  on acpi0
vmgenc0:  on acpi0
acpi_syscontainer1:  port 0xcd8-0xce3 on acpi0
acpi_syscontainer2:  port 0x620-0x62f on acpi0
acpi_syscontainer3:  port 0xcc0-0xcd7 on acpi0
atkbdc0:  port 0x60,0x64 irq 1 on acpi0
atkbd0:  irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0:  irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
attimer0:  at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounters tick every 10.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ugen6.1:  at usbus6
ugen0.1:  at usbus0
uhub0 on usbus6
uhub0:  on usbus6
ugen5.1:  at usbus5
uhub1 on usbus0
uhub1:  on usbus0
ugen7.1:  at usbus7
uhub2 on usbus5
uhub2:  on usbus5
ugen3.1:  at usbus3
uhub3 on usbus7
uhub3:  on usbus7
uhub4 on usbus3
uhub4:  on usbus3
ugen4.1:  at usbus4
ugen1.1:  at usbus1
uhub5 on usbus4
uhub5:  on usbus4
ugen2.1:  at usbus2
uhub6 on usbus1
uhub6:  on usbus1
uhub7 on usbus2
uhub7:  on usbus2
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Root mount waiting for: usbus0 usbus1 usbus2 usbus3 usbus4 usbus5 usbus6 usbus7 CAM
da0 at vtscsi0 bus 0 scbus0 target 0 lun 0
da0:  Fixed Direct Access SPC-3 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 122880MB (251658240 512 byte sectors)
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub7: 2 ports with 2 removable, self powered
Root mount waiting for: usbus3 usbus7
Root mount waiting for: usbus3 usbus7
uhub3: 6 ports with 6 removable, self powered
uhub4: 6 ports with 6 removable, self powered
mountroot: waiting for device /dev/gpt/rootfs...
vtcon0:  on virtio_pci2
ichsmb0:  port 0xd200-0xd23f irq 16 at device 31.3 on pci0
smbus0:  on ichsmb0
lo0: link state changed to UP
vtnet0: link state changed to UP
vlan0: changing name to 'vtnet0_vlan10'
vlan1: changing name to 'vtnet0_vlan20'
vlan2: changing name to 'vtnet0_vlan30'
vlan3: changing name to 'vtnet0_vlan40'
vtnet1: link state changed to UP
pflog0: permanently promiscuous mode enabled
WARNING: attempt to domain_add(netgraph) after domainfinalize()
kernel trap 1 with interrupts disabled


Fatal trap 1: privileged instruction fault while in kernel mode
cpuid = 1; apic id = 01
instruction pointer = 0x20:0xffffffff8120d8d0
stack pointer         = 0x28:0xfffffe00c5f79a38
frame pointer         = 0x28:0xfffffe00c5f79b00
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 18 (syncer)
trap number = 1
panic: privileged instruction fault
cpuid = 1
time = 1658746079
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c5f79850
vpanic() at vpanic+0x17f/frame 0xfffffe00c5f798a0
panic() at panic+0x43/frame 0xfffffe00c5f79900
trap_fatal() at trap_fatal+0x385/frame 0xfffffe00c5f79960
calltrap() at calltrap+0x8/frame 0xfffffe00c5f79960
--- trap 0x1, rip = 0xffffffff8120d8d0, rsp = 0xfffffe00c5f79a38, rbp = 0xfffffe00c5f79b00 ---
lapic_handle_timer() at lapic_handle_timer/frame 0xfffffe00c5f79b00
cam_periph_release_locked() at cam_periph_release_locked+0x13/frame 0xfffffe00c5f79b20
dastrategy() at dastrategy+0x82/frame 0xfffffe00c5f79b60
g_disk_start() at g_disk_start+0x314/frame 0xfffffe00c5f79bc0
g_io_request() at g_io_request+0x1d9/frame 0xfffffe00c5f79bf0
g_io_request() at g_io_request+0x1d9/frame 0xfffffe00c5f79c20
g_io_request() at g_io_request+0x1d9/frame 0xfffffe00c5f79c50
ufs_strategy() at ufs_strategy+0x83/frame 0xfffffe00c5f79c80
bufstrategy() at bufstrategy+0x36/frame 0xfffffe00c5f79cb0
bufwrite() at bufwrite+0x1dc/frame 0xfffffe00c5f79cf0
cluster_wbuild() at cluster_wbuild+0x6d5/frame 0xfffffe00c5f79d90
ffs_syncvnode() at ffs_syncvnode+0x3a9/frame 0xfffffe00c5f79e30
ffs_fsync() at ffs_fsync+0x22/frame 0xfffffe00c5f79e70
sched_sync() at sched_sync+0x47e/frame 0xfffffe00c5f79ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00c5f79f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c5f79f30
--- trap 0x80c6831d, rip = 0xffffffff80c2b90f, rsp = 0, rbp = 0xfffff80001a47548 ---
mi_startup() at mi_startup+0xdf/frame 0xfffff80001a47548
??() at 0xfffff80004b74548/frame 0xfffff80001a47a90
??() at 0xfffff80001a47548/frame 0xfffff800017f2000
??() at 0xfffff80001a47a90/frame 0xfffff800017f2548
??() at 0xfffff800017f2000/frame 0xfffff800017f2a90
??() at 0xfffff800017f2548/frame 0xfffff800017f1000
??() at 0xfffff800017f2a90/frame 0xfffff800017f1548
??() at 0xfffff800017f1000/frame 0xfffff800017f1a90
??() at 0xfffff800017f1548/frame 0xfffff80001a47000
??() at 0xfffff800017f1a90/frame 0xfffff80001744000
??() at 0xfffff80001a47000/frame 0xfffff80001744548
??() at 0xfffff80001744000/frame 0xfffff80001744a90
??() at 0xfffff80001744548/frame 0xfffff80001743000
??() at 0xfffff80001744a90/frame 0xfffff80001743548
??() at 0xfffff80001743000/frame 0xfffff80001743a90
??() at 0xfffff80001743548/frame 0xfffff80001742000
??() at 0xfffff80001743a90/frame 0xfffff80001742548
??() at 0xfffff80001742000/frame 0xffffffff81e9e748
??() at 0xfffff80001742548
KDB: enter: panic

[DKSF]

I've not been able to find the root cause of the issue and get OPNsense stable. Tried pfSense which has been stable and running without any issues smoothly in the exact same HW setup, same modules, same configuration.