26.7: kernel panic in bridge_input() via Suricata netmap on bridge0, not 26.1

[yoavain]

Setup

Intel Core 3 N355 mini PC, 16 GB RAM, 8x Intel I226-V (igc0 to igc7), ZFS root, UEFI boot.

• igc4 is the WAN, PPPoE.
• The remaining seven ports are members of bridge0, a flat LAN at 192.168.68.1/24.
• Suricata (IPS mode) is bound to bridge0, which has no native netmap support, so netmap attaches a generic (emulated) adapter.

All offloading is disabled in config as the netmap documentation requires:

Code Select Expand

<disablechecksumoffloading>1</disablechecksumoffloading>
<disablesegmentationoffloading>1</disablesegmentationoffloading>
<disablelargereceiveoffloading>1</disablelargereceiveoffloading>
<disablevlanhwfilter>1</disablevlanhwfilter>
and this is reflected at runtime:

Code Select Expand

igc0: options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
os-intel-microcode is not installed, so the microcode issue noted in the release announcement does not apply.

This configuration is stable on 26.1.11_10 (FreeBSD 15.0) and has been for months. bridge0 currently shows options=100000<NETMAP> with Suricata attached and running. On 26.7.1_1 (FreeBSD 15.1) the same configuration panics.

Symptom

The system boots fully: all interfaces come up, PPPoE authenticates and obtains an address, Unbound and the web GUI start, and the console banner prints. Roughly 107 seconds later, once Suricata finishes loading rules and attaches netmap to bridge0, the kernel panics. It then reboots and repeats, which from the outside looks like a boot loop but is not one.

Reproduced twice, at identical uptime, on consecutive boots.

Panic

Code Select Expand

[107] 147.957436 [ 321] generic_netmap_register  Emulated adapter for bridge0 activated
[107]
[107]
[107] Fatal trap 12: page fault while in kernel mode
[107] cpuid = 4; apic id = 08
[107] fault virtual address= 0x18
[107] fault code        = supervisor read data, page not present
[107] instruction pointer = 0x20:0xffffffff82eda008
[108] stack pointer            = 0x28:0xfffffe011d027880
[108] frame pointer            = 0x28:0xfffffe011d0278e0
[108] code segment      = base 0x0, limit 0xfffff, type 0x1b
[108]                  = DPL 0, pres 1, long 1, def32 0, gran 1
[108] processor eflags  = interrupt enabled, resume, IOPL = 0
[108] current process    = 60970 (W#01-bridge0)
[108] rdi: 0000000000000028 rsi: fffff80225320376 rdx: 0000000000000000
[108] rcx: 0000000000000000  r8: 00000000bc571c01  r9: 0000000000000155
[108] rax: 0000000004000201 rbx: fffff80225320300 rbp: fffffe011d0278e0
[108] r10: 00000000b183d5d8 r11: 00000000525d7235 r12: 0000000000000000
[108] r13: fffff800241e1400 r14: fffff800247fb800 r15: fffff800015e5000
[108] trap number        = 12
[108] panic: page fault
[108] cpuid = 4
[108] time = 1784838148
[108] KDB: stack backtrace:
[108] db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe011d0275d0
[108] vpanic() at vpanic+0x136/frame 0xfffffe011d027700
[108] panic() at panic+0x43/frame 0xfffffe011d027760
[108] trap_pfault() at trap_pfault+0x37d/frame 0xfffffe011d0277b0
[108] calltrap() at calltrap+0x8/frame 0xfffffe011d0277b0
[108] --- trap 0xc, rip = 0xffffffff82eda008, rsp = 0xfffffe011d027880, rbp = 0xfffffe011d0278e0 ---
[108] bridge_input() at bridge_input+0xd8/frame 0xfffffe011d0278e0
[108] ether_nh_input() at ether_nh_input+0x2b2/frame 0xfffffe011d027940
[108] netisr_dispatch_src() at netisr_dispatch_src+0x9f/frame 0xfffffe011d027990
[108] ether_input() at ether_input+0x56/frame 0xfffffe011d0279e0
[108] nm_os_send_up() at nm_os_send_up+0x1e/frame 0xfffffe011d027a00
[108] netmap_send_up() at netmap_send_up+0x4e/frame 0xfffffe011d027a50
[108] netmap_txsync_to_host() at netmap_txsync_to_host+0x72/frame 0xfffffe011d027ad0
[108] netmap_ioctl() at netmap_ioctl+0x1a6/frame 0xfffffe011d027bb0
[108] freebsd_netmap_ioctl() at freebsd_netmap_ioctl+0x79/frame 0xfffffe011d027bf0
[108] devfs_ioctl() at devfs_ioctl+0xcb/frame 0xfffffe011d027c40
[108] vn_ioctl() at vn_ioctl+0xc8/frame 0xfffffe011d027cb0
[108] devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe011d027cd0
[108] kern_ioctl() at kern_ioctl+0x286/frame 0xfffffe011d027d40
[108] sys_ioctl() at sys_ioctl+0x101/frame 0xfffffe011d027e00
[108] amd64_syscall() at amd64_syscall+0x126/frame 0xfffffe011d027f30
[108] fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe011d027f30
[108] --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x82cff826a, rsp = 0x82eb04e48, rbp = 0x82eb04e70 ---
[108] KDB: enter: panic
---<<BOOT>>---
The second occurrence is identical in structure, including bridge_input+0xd8, on a different CPU (cpuid = 0, process 80641 (W#01-bridge0)). The absolute instruction pointer differs (0xffffffff8277f008) because the module load address changed between boots. The netmap attach immediately preceding it:

Code Select Expand

[107] 301.683015 [1087] generic_netmap_dtor      Emulated netmap adapter for bridge0 destroyed
[107] bridge0: permanently promiscuous mode enabled
[107] 301.693515 [1182] generic_netmap_attach    Emulated adapter for bridge0 created (prev was NULL)
[107] 301.754355 [ 321] generic_netmap_register  Emulated adapter for bridge0 activated
Kernel:

Code Select Expand

FreeBSD 15.1-RELEASE-p1 stable/26.7-n283674-12334a596709 SMP amd64
Textdumps

savecore captured both to /var/crash/textdump.tar.15 and /var/crash/textdump.tar.16. Available on request.

[franco]

> This configuration is stable on 26.1.11_10 (FreeBSD 15.0) and has been for months.

Yeah but 26.1 is 14.3, not 15.x.  Looking at a nice regression here.

Please add a ticket on GitHub to track this.


Cheers,
Franco

[yoavain]

Follow-up: possible mechanism

FreeBSD 15.1 enables VLAN_HWTAGGING on bridge0, and it cannot be cleared.

Code Select Expand

# ifconfig -m bridge0 | grep -E 'capabilities|options='
        options=10<VLAN_HWTAGGING>
        capabilities=10<VLAN_HWTAGGING>

# ifconfig bridge0 -vlanhwtag
ifconfig: -vlanhwtag: Invalid argument
None of the seven bridge members have it enabled:

Code Select Expand

igc0: options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
igc1: options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
igc2: options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
igc3: options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
igc5: options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
igc6: options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
igc7: options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
Same box on 26.1 (FreeBSD 15.0), same members, same config, Suricata stopped:

Code Select Expand

bridge0: options=0
So bridge0 now advertises and enables a capability that no member enables, and SIOCSIFCAP rejects clearing it. The netmap documentation requires VLAN hardware tagging to be disabled, which makes that prerequisite unsatisfiable on a bridge under 15.1.

This also explains a boot log line that appears on both versions and is harmless on 15.0:

Code Select Expand

rc.bootup: The command `/sbin/ifconfig 'bridge0' -vlanhwtag' failed to execute
I have not traced further than this and cannot say whether the capability derivation or the netmap path is at fault, but the 15.0 to 15.1 change in bridge0 capabilities looks like the place to start.

[yoavain]

Opened a ticket
https://github.com/opnsense/src/issues/305

[franco]

Thank you. Will look into it.