Topics

1

22.1 Legacy Series / Sudden kernel trap after domain_add(netgraph) after domainfinalize()

« on: July 26, 2022, 08:00:11 pm »

Hi - I'm running into kernel traps that I need some help with diagnosing. IPv6 is turned off on all interfaces. Any suggestions on where to look? This is a newly installed clean updated 22.1.10.

The trigger seems to be this:

Code: [Select]

WARNING: attempt to domain_add(netgraph) after domainfinalize()
kernel trap 1 with interrupts disabled
Any pointers?

The full report from dmesg is as follows.

Code: [Select]

FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(efifb): resolution 1024x768
CPU: Intel(R) Celeron(R) N5105 @ 2.00GHz (1996.83-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906c0  Family=0x6  Model=0x9c  Stepping=0
  Features=0x1f83fbff
  Features2=0xcff8a223
  AMD Features=0x28100800
  AMD Features2=0x101
  Structured Extended Features=0x21940283
  Structured Extended Features2=0x18400124
  Structured Extended Features3=0xac000400
  XSAVE Features=0xf
  IA32_ARCH_CAPS=0x6b
  AMD Extended Feature Extensions ID EBX=0x100d000
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
Hypervisor: Origin = "KVMKVMKVM"
real memory  = 8589934592 (8192 MB)
avail memory = 8235425792 (7853 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table:
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0  irqs 0-23
Launching APs: 1
random: entropy device external interface
wlan: mac acl policy registered
kbd1 at kbdmux0
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
kvmclock0:
Timecounter "kvmclock" frequency 1000000000 Hz quality 975
kvmclock0: registered as a time-of-day clock, resolution 0.000001s
efirtc0:
efirtc0: registered as a time-of-day clock, resolution 1.000000s
aesni0:
acpi0:
acpi0: Power Button (fixed)
cpu0:  numa-domain 0 on acpi0
atrtc0:  port 0x70-0x77 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pcib0:  port 0xcf8-0xcff on acpi0
pci0:  on pcib0
vgapci0:  mem 0xc0000000-0xc0ffffff,0xc224b000-0xc224bfff at device 1.0 on pci0
vgapci0: Boot video device
uhci0:  port 0xd300-0xd31f irq 16 at device 26.0 on pci0
usbus0 on uhci0
usbus0: 12Mbps Full Speed USB v1.0
uhci1:  port 0xd2e0-0xd2ff irq 17 at device 26.1 on pci0
usbus1 on uhci1
usbus1: 12Mbps Full Speed USB v1.0
uhci2:  port 0xd2c0-0xd2df irq 18 at device 26.2 on pci0
usbus2 on uhci2
usbus2: 12Mbps Full Speed USB v1.0
ehci0:  mem 0xc224a000-0xc224afff irq 19 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
hdac0:  mem 0xc2240000-0xc2243fff irq 16 at device 27.0 on pci0
pcib1:  mem 0xc2249000-0xc2249fff irq 16 at device 28.0 on pci0
pcib1: failed to allocate initial I/O port window: 0xd000-0xdfff
pci1:  on pcib1
pcib2:  mem 0xc2248000-0xc2248fff irq 16 at device 28.1 on pci0
pci2:  on pcib2
pcib3:  mem 0xc2247000-0xc2247fff irq 16 at device 28.2 on pci0
pci3:  on pcib3
pcib4:  mem 0xc2246000-0xc2246fff irq 16 at device 28.3 on pci0
pci4:  on pcib4
uhci3:  port 0xd2a0-0xd2bf irq 16 at device 29.0 on pci0
usbus4 on uhci3
usbus4: 12Mbps Full Speed USB v1.0
uhci4:  port 0xd280-0xd29f irq 17 at device 29.1 on pci0
usbus5 on uhci4
usbus5: 12Mbps Full Speed USB v1.0
uhci5:  port 0xd260-0xd27f irq 18 at device 29.2 on pci0
usbus6 on uhci5
usbus6: 12Mbps Full Speed USB v1.0
ehci1:  mem 0xc2245000-0xc2245fff irq 19 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
usbus7: 480Mbps High Speed USB v2.0
pcib5:  at device 30.0 on pci0
pci5:  on pcib5
pcib6:  mem 0xc1800000-0xc18000ff irq 21 at device 1.0 on pci5
pci6:  on pcib6
virtio_pci0:  port 0x9080-0x90bf mem 0x800014000-0x800017fff irq 20 at device 3.0 on pci6
vtballoon0:  on virtio_pci0
virtio_pci1:  port 0x9040-0x907f mem 0xc1604000-0xc1604fff,0x800010000-0x800013fff irq 22 at device 5.0 on pci6
vtscsi0:  on virtio_pci1
virtio_pci2:  port 0x9000-0x903f mem 0xc1603000-0xc1603fff,0x80000c000-0x80000ffff irq 21 at device 8.0 on pci6
virtio_pci3:  port 0x9100-0x911f mem 0xc1602000-0xc1602fff,0x800008000-0x80000bfff irq 23 at device 18.0 on pci6
vtnet0:  on virtio_pci3
vtnet0: Ethernet address: 1a:a9:c8:38:17:22
vtnet0: netmap queues/slots: TX 1/256, RX 1/128
000.000121 [ 450] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=128
virtio_pci4:  port 0x90e0-0x90ff mem 0xc1601000-0xc1601fff,0x800004000-0x800007fff irq 20 at device 19.0 on pci6
vtnet1:  on virtio_pci4
vtnet1: Ethernet address: 00:e0:4c:92:86:72
vtnet1: netmap queues/slots: TX 1/256, RX 1/128
000.000122 [ 450] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=128
virtio_pci5:  port 0x90c0-0x90df mem 0xc1600000-0xc1600fff,0x800000000-0x800003fff irq 21 at device 20.0 on pci6
vtnet2:  on virtio_pci5
vtnet2: Ethernet address: ce:7a:38:20:57:21
vtnet2: netmap queues/slots: TX 1/256, RX 1/128
000.000123 [ 450] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=128
pcib7:  mem 0xc1801000-0xc18010ff irq 22 at device 2.0 on pci5
pci7:  on pcib7
pcib8:  mem 0xc1802000-0xc18020ff irq 23 at device 3.0 on pci5
pci8:  on pcib8
pcib9:  mem 0xc1803000-0xc18030ff irq 20 at device 4.0 on pci5
pci9:  on pcib9
isab0:  at device 31.0 on pci0
isa0:  on isab0
ahci0:  port 0xd240-0xd25f mem 0xc2244000-0xc2244fff irq 16 at device 31.2 on pci0
ahci0: AHCI v1.00 with 6 1.5Gbps ports, Port Multiplier not supported
ahcich0:  at channel 0 on ahci0
ahcich1:  at channel 1 on ahci0
ahcich2:  at channel 2 on ahci0
ahcich3:  at channel 3 on ahci0
ahcich4:  at channel 4 on ahci0
ahcich5:  at channel 5 on ahci0
acpi_syscontainer0:  on acpi0
vmgenc0:  on acpi0
acpi_syscontainer1:  port 0xcd8-0xce3 on acpi0
acpi_syscontainer2:  port 0x620-0x62f on acpi0
acpi_syscontainer3:  port 0xcc0-0xcd7 on acpi0
atkbdc0:  port 0x60,0x64 irq 1 on acpi0
atkbd0:  irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0:  irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.
psm0: model IntelliMouse Explorer, device ID 4
attimer0:  at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounters tick every 10.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ugen6.1:  at usbus6
ugen0.1:  at usbus0
uhub0 on usbus6
uhub0:  on usbus6
ugen5.1:  at usbus5
uhub1 on usbus0
uhub1:  on usbus0
ugen7.1:  at usbus7
uhub2 on usbus5
uhub2:  on usbus5
ugen3.1:  at usbus3
uhub3 on usbus7
uhub3:  on usbus7
uhub4 on usbus3
uhub4:  on usbus3
ugen4.1:  at usbus4
ugen1.1:  at usbus1
uhub5 on usbus4
uhub5:  on usbus4
ugen2.1:  at usbus2
uhub6 on usbus1
uhub6:  on usbus1
uhub7 on usbus2
uhub7:  on usbus2
Trying to mount root from ufs:/dev/gpt/rootfs [rw]...
Root mount waiting for: usbus0 usbus1 usbus2 usbus3 usbus4 usbus5 usbus6 usbus7 CAM
da0 at vtscsi0 bus 0 scbus0 target 0 lun 0
da0:  Fixed Direct Access SPC-3 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 122880MB (251658240 512 byte sectors)
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub7: 2 ports with 2 removable, self powered
Root mount waiting for: usbus3 usbus7
Root mount waiting for: usbus3 usbus7
uhub3: 6 ports with 6 removable, self powered
uhub4: 6 ports with 6 removable, self powered
mountroot: waiting for device /dev/gpt/rootfs...
vtcon0:  on virtio_pci2
ichsmb0:  port 0xd200-0xd23f irq 16 at device 31.3 on pci0
smbus0:  on ichsmb0
lo0: link state changed to UP
vtnet0: link state changed to UP
vlan0: changing name to 'vtnet0_vlan10'
vlan1: changing name to 'vtnet0_vlan20'
vlan2: changing name to 'vtnet0_vlan30'
vlan3: changing name to 'vtnet0_vlan40'
vtnet1: link state changed to UP
pflog0: permanently promiscuous mode enabled
WARNING: attempt to domain_add(netgraph) after domainfinalize()
kernel trap 1 with interrupts disabled


Fatal trap 1: privileged instruction fault while in kernel mode
cpuid = 1; apic id = 01
instruction pointer = 0x20:0xffffffff8120d8d0
stack pointer         = 0x28:0xfffffe00c5f79a38
frame pointer         = 0x28:0xfffffe00c5f79b00
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 18 (syncer)
trap number = 1
panic: privileged instruction fault
cpuid = 1
time = 1658746079
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c5f79850
vpanic() at vpanic+0x17f/frame 0xfffffe00c5f798a0
panic() at panic+0x43/frame 0xfffffe00c5f79900
trap_fatal() at trap_fatal+0x385/frame 0xfffffe00c5f79960
calltrap() at calltrap+0x8/frame 0xfffffe00c5f79960
--- trap 0x1, rip = 0xffffffff8120d8d0, rsp = 0xfffffe00c5f79a38, rbp = 0xfffffe00c5f79b00 ---
lapic_handle_timer() at lapic_handle_timer/frame 0xfffffe00c5f79b00
cam_periph_release_locked() at cam_periph_release_locked+0x13/frame 0xfffffe00c5f79b20
dastrategy() at dastrategy+0x82/frame 0xfffffe00c5f79b60
g_disk_start() at g_disk_start+0x314/frame 0xfffffe00c5f79bc0
g_io_request() at g_io_request+0x1d9/frame 0xfffffe00c5f79bf0
g_io_request() at g_io_request+0x1d9/frame 0xfffffe00c5f79c20
g_io_request() at g_io_request+0x1d9/frame 0xfffffe00c5f79c50
ufs_strategy() at ufs_strategy+0x83/frame 0xfffffe00c5f79c80
bufstrategy() at bufstrategy+0x36/frame 0xfffffe00c5f79cb0
bufwrite() at bufwrite+0x1dc/frame 0xfffffe00c5f79cf0
cluster_wbuild() at cluster_wbuild+0x6d5/frame 0xfffffe00c5f79d90
ffs_syncvnode() at ffs_syncvnode+0x3a9/frame 0xfffffe00c5f79e30
ffs_fsync() at ffs_fsync+0x22/frame 0xfffffe00c5f79e70
sched_sync() at sched_sync+0x47e/frame 0xfffffe00c5f79ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00c5f79f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c5f79f30
--- trap 0x80c6831d, rip = 0xffffffff80c2b90f, rsp = 0, rbp = 0xfffff80001a47548 ---
mi_startup() at mi_startup+0xdf/frame 0xfffff80001a47548
??() at 0xfffff80004b74548/frame 0xfffff80001a47a90
??() at 0xfffff80001a47548/frame 0xfffff800017f2000
??() at 0xfffff80001a47a90/frame 0xfffff800017f2548
??() at 0xfffff800017f2000/frame 0xfffff800017f2a90
??() at 0xfffff800017f2548/frame 0xfffff800017f1000
??() at 0xfffff800017f2a90/frame 0xfffff800017f1548
??() at 0xfffff800017f1000/frame 0xfffff800017f1a90
??() at 0xfffff800017f1548/frame 0xfffff80001a47000
??() at 0xfffff800017f1a90/frame 0xfffff80001744000
??() at 0xfffff80001a47000/frame 0xfffff80001744548
??() at 0xfffff80001744000/frame 0xfffff80001744a90
??() at 0xfffff80001744548/frame 0xfffff80001743000
??() at 0xfffff80001744a90/frame 0xfffff80001743548
??() at 0xfffff80001743000/frame 0xfffff80001743a90
??() at 0xfffff80001743548/frame 0xfffff80001742000
??() at 0xfffff80001743a90/frame 0xfffff80001742548
??() at 0xfffff80001742000/frame 0xffffffff81e9e748
??() at 0xfffff80001742548
KDB: enter: panic