VLAN support on bridges from FreeBSD 15.x onwards, impact when?

[franco]

If you patch https://github.com/opnsense/core/commit/6de81ecaf17ed5ba1 beforehand it should work.

[br41n]

Thanks franco.

After applying the recommended patch beforehand and afterwards the patch from Cedrik it is working.

[ArthurK.CICS]

Bug report — CARP VIP on VLAN-over-bridge

We are running 137513c on 26.7.1_1 per the instructions earlier in this thread — the feature itself works great for us (bridged SFP+ fabric in a virtualized HA pair). With CARP on top of the VLAN-on-bridge interfaces, however, we hit what looks like a gap in if_bridge's local-delivery path:

ARP and CARP advertisements work, but unicast to the VIP's virtual MAC is never delivered.

Setup

• bridge0 over five physical members (VM with PCIe passthrough NICs, but nothing here is virtualization-specific).
• L3 untagged directly on bridge0 (assigned interface), plus tagged VLANs vlan0.10 ... vlan0.19 with parent bridge0 (per the patch), each assigned with a static IPv4 and a CARP VIP (vhid = VLAN tag).
• Second node with an identical layout; CARP advertisements cross a bridged peer link.

What works

• L3 on the VLAN-on-bridge interfaces: unicast to the interface's own address/MAC is fine.
• CARP election per VLAN: advertisements are sent and received tagged, MASTER/BACKUP negotiate correctly, failover and preempt work.
• ARP for the VIP: clients resolve it and learn the virtual MAC (00:00:5e:00:01:<vhid>).
• A CARP VIP configured on bridge0 itself (the untagged side): fully functional including unicast — this one is not affected.

The bug

Unicast frames addressed to the CARP virtual MAC of a VLAN-on-bridge interface are never delivered to that interface's IP stack. Observed on the box itself:

Code Select Expand


# tcpdump -i bridge0 -e -nn ether host 00:00:5e:00:01:0a

xx:xx:xx:xx:xx:xx > 00:00:5e:00:01:0a, 802.1Q, vlan 10, IPv4,
    10.0.10.53 > 10.0.10.3: ICMP echo request ...
(frames arrive continuously)

# tcpdump -i vlan0.10 -nn host 10.0.10.3

(nothing - ever)

So the frames reach bridge0 with the correct tag, but the bridge treats unicast to the VLAN child's virtual MAC as unknown unicast (floods it to the members) instead of delivering it upward. The second HA node shows the same split from its own vantage point: it can ping the untagged VIP on bridge0, but none of the tagged VIPs — with identical CARP config (unique vhids, verified) and a healthy MASTER/BACKUP election on every vhid. The client side shows the classic signature: the VIP's ARP resolves, the neighbor entry then hangs in PROBE, pings to the VIP are 100% loss while pings to the interface address on the same VLAN work — and the firewall log shows the flows as passed, so everything looks healthy.

Suspected cause

The bridge's "is this ours?" check on the RX path appears to match member MACs, the bridge's own MAC, and CARP MACs bound to the bridge interface itself — but not CARP vhids attached to VLAN interfaces stacked on the bridge. The untagged case working while every tagged case fails is consistent with that.

Repro

• Apply 137513c on 26.7.1, create bridge0 with at least one member, assign and address it.
• Create vlan0.10 with parent bridge0, assign, address it (e.g. 10.0.10.1/24).
• Add a CARP VIP 10.0.10.3/24, vhid 10, on that VLAN interface.
• From any host on VLAN 10: ping 10.0.10.1 → works; ping 10.0.10.3 → ARP resolves, then 100% loss. tcpdump as above shows the frames dying inside the bridge.

Impact

CARP VIPs on VLANs-over-bridge cannot serve as gateway addresses at all, which silently breaks HA-gateway designs on this topology. It is doubly deceptive because everything observable (election, ARP, firewall logs) looks fine.

Is this addressed in the native FreeBSD 15 / 27.1 implementation, or does it need a separate if_bridge fix (consulting the CARP MACs of stacked VLAN devices in the local-delivery decision)? Happy to test patches — we can reproduce this on demand.

[Monviech (Cedrik)]

Please prove it with jails:

https://github.com/opnsense/core/pull/10584#issuecomment-5025745485

Example in this comment. You can extend my baseline jail scripts, if you have a reproducible case please share the updated jail scripts.

Thank you.

[ArthurK.CICS]

Reproduced in jails on a clean FreeBSD 15.1-RELEASE (GENERIC). No OPNsense code involved, so this is not specific to the 137513c backport — the native implementation behaves the same way, and 27.1 would inherit it.

I extended your setup script with CARP. Jail b gets a VIP on its vlan3 (over the bridge) and a second VIP untagged on bridge0 itself; jail c gets a VIP on its vlan3 (directly on the epair) as control. Fresh 15.1 VM, kldload carp, nothing else.

Pings from jail a:

• interface addresses (172.16.2.2, 172.16.3.2, 172.16.2.3): all fine
• VIP on vlan-over-epair, jail c (172.16.2.23): fine
• VIP untagged on bridge0 itself, jail b (172.16.3.13): fine
• VIP on vlan3 over bridge0, jail b (172.16.2.13): ARP resolves, then 100% loss. The echo requests show up on epair1b and on bridge0, never on vlan3, and netstat's ICMP input counter in jail b stays flat.

carp_jail_setup.sh (your setup + CARP)

Code Select Expand

#!/bin/sh

# Extension of Monviech's baseline jail scripts (opnsense/core PR #10584)

# to reproduce: CARP VIP on a VLAN-over-bridge interface does not receive unicast.

#

# jail a = client (tagged vlan3 + untagged leg)

# jail b = DUT: vlan3 stacked on an internal bridge0

#          -> CARP VIP on vlan3   (vhid 13, 172.16.2.13)  = the bug case

#          -> CARP VIP on bridge0 (vhid 33, 172.16.3.13)  = control, untagged on the bridge itself

# jail c = control: vlan3 directly on the epair

#          -> CARP VIP on vlan3   (vhid 23, 172.16.2.23)  = control, no bridge involved

kldload -n carp

ifconfig epair0 create
ifconfig epair1 create
ifconfig epair2 create

ifconfig bridge0 create \
    addm epair0a \
    addm epair1a \
    addm epair2a \
    up

ifconfig epair0a up
ifconfig epair1a up
ifconfig epair2a up

jail -c name=a path=/ persist vnet allow.raw_sockets=1
jail -c name=b path=/ persist vnet allow.raw_sockets=1
jail -c name=c path=/ persist vnet allow.raw_sockets=1

ifconfig epair0b vnet a
ifconfig epair1b vnet b
ifconfig epair2b vnet c

# jail a - client

jexec a ifconfig epair0b inet 172.16.3.1/24 up
jexec a ifconfig vlan3 create vlan 3 vlandev epair0b
jexec a ifconfig vlan3 inet 172.16.2.1/24 up

# jail b - DUT: vlan on bridge, CARP on both layers

jexec b ifconfig epair1b up
jexec b ifconfig bridge0 create addm epair1b up
jexec b ifconfig bridge0 inet 172.16.3.2/24
jexec b ifconfig vlan3 create vlan 3 vlandev bridge0
jexec b ifconfig vlan3 inet 172.16.2.2/24 up
jexec b ifconfig vlan3 vhid 13 pass testpass alias 172.16.2.13/32
jexec b ifconfig bridge0 vhid 33 pass testpass alias 172.16.3.13/32

# jail c - control: vlan directly on the epair, CARP on it

jexec c ifconfig epair2b up
jexec c ifconfig vlan3 create vlan 3 vlandev epair2b
jexec c ifconfig vlan3 inet 172.16.2.3/24 up
jexec c ifconfig vlan3 vhid 23 pass testpass alias 172.16.2.23/32

carp_jail_test.sh

Code Select Expand

#!/bin/sh

# Test companion to carp_jail_setup.sh - CARP VIP on VLAN-over-bridge repro.

# Expected on an affected kernel:

#   interface addresses        : all reachable

#   VIP on vlan-over-epair (c) : reachable            (control 1)

#   VIP untagged on bridge (b) : reachable            (control 2)

#   VIP on vlan-over-bridge (b): ARP resolves, 100% loss - frames visible on

#                                the bridge member and bridge0, never on vlan3,

#                                ICMP input counter on b does not move.  (BUG)

LOG=${1:-carp-vlan-bridge-test.log}
TMP=/tmp/carp-vlan-bridge-test.$$

wait_master() { # jail iface
    n=0
    while [ $n -lt 15 ]; do
        jexec "$1" ifconfig "$2" | grep -q "carp: MASTER" && return 0
        sleep 1; n=$((n+1))
    done
    echo "WARNING: $1/$2 never reached MASTER"
    jexec "$1" ifconfig "$2"
}

icmp_in_echo() { # jail - echo requests received (Input histogram: echo)
    jexec "$1" netstat -s -p icmp | sed -n '/Input histogram/,/^[^[:space:]]/p' \
        | awk '/echo:/{print $2; exit}'
}

{
    echo "=== CARP election state ==="
    wait_master b vlan3
    wait_master b bridge0
    wait_master c vlan3
    jexec b ifconfig vlan3
    jexec b ifconfig bridge0
    jexec c ifconfig vlan3

    echo
    echo "=== baseline: interface addresses ==="
    jexec a ping -c 2 -t 4 172.16.2.2 || true
    jexec a ping -c 2 -t 4 172.16.3.2 || true
    jexec a ping -c 2 -t 4 172.16.2.3 || true
   
    echo
    echo "=== CONTROL 1: CARP VIP on vlan-over-EPAIR (c, 172.16.2.23, vhid 23) ==="
    jexec a ping -c 2 -t 4 172.16.2.23 || true
   
    echo
    echo "=== CONTROL 2: CARP VIP UNTAGGED on the bridge itself (b, 172.16.3.13, vhid 33) ==="
    jexec a ping -c 2 -t 4 172.16.3.13 || true
   
    echo
    echo "=== BUG CASE: CARP VIP on vlan-over-BRIDGE (b, 172.16.2.13, vhid 13) ==="
    BEFORE=$(icmp_in_echo b)
   
    jexec b tcpdump -nlei epair1b ether host 00:00:5e:00:01:0d >"${TMP}.member" 2>&1 &
    PID0=$!
    jexec b tcpdump -nlei bridge0 ether host 00:00:5e:00:01:0d >"${TMP}.bridge" 2>&1 &
    PID1=$!
    jexec b tcpdump -nlei vlan3 host 172.16.2.13 >"${TMP}.vlan" 2>&1 &
    PID2=$!
   
    sleep 1
    jexec a ping -c 3 -t 6 172.16.2.13 || true
    sleep 1
   
    kill "$PID0" "$PID1" "$PID2" 2>/dev/null
    wait "$PID0" "$PID1" "$PID2" 2>/dev/null
   
    AFTER=$(icmp_in_echo b)
   
    echo
    echo "--- client (a) neighbor table: virtual MAC resolved? ---"
    jexec a arp -an | grep 172.16.2.13 || echo "(no ARP entry)"
   
    echo
    echo "--- ICMP echo requests received by b's stack: before=$BEFORE after=$AFTER ---"
   
    echo
    echo "--- tcpdump b:epair1b, bridge member (frames TO the virtual MAC 00:00:5e:00:01:0d) ---"
    cat "${TMP}.member"
   
    echo
    echo "--- tcpdump b:bridge0 (frames TO the virtual MAC 00:00:5e:00:01:0d) ---"
    cat "${TMP}.bridge"
   
    echo
    echo "--- tcpdump b:vlan3 (host 172.16.2.13) ---"
    cat "${TMP}.vlan"

} >"$LOG" 2>&1

rm -f "${TMP}.member" "${TMP}.bridge" "${TMP}.vlan"
cat "$LOG"

carp_jail_teardown.sh

Code Select Expand

#!/bin/sh
for j in a b c; do
    jail -r "$j" 2>/dev/null
done
for i in epair0a epair1a epair2a; do
    ifconfig "$i" destroy 2>/dev/null
done
ifconfig bridge0 destroy 2>/dev/null

Log excerpts (full log available if useful):

Code Select Expand

$ uname -a
FreeBSD carp-jailtest 15.1-RELEASE FreeBSD 15.1-RELEASE releng/15.1-n283562-96841ea08dcf GENERIC amd64

=== CONTROL 1: CARP VIP on vlan-over-EPAIR (c, 172.16.2.23, vhid 23) ===
2 packets transmitted, 2 packets received, 0.0% packet loss

=== CONTROL 2: CARP VIP UNTAGGED on the bridge itself (b, 172.16.3.13, vhid 33) ===
2 packets transmitted, 2 packets received, 0.0% packet loss

=== BUG CASE: CARP VIP on vlan-over-BRIDGE (b, 172.16.2.13, vhid 13) ===
3 packets transmitted, 0 packets received, 100.0% packet loss

--- client (a) neighbor table: virtual MAC resolved? ---
? (172.16.2.13) at 00:00:5e:00:01:0d on vlan3 expires in 1193 seconds [vlan]

--- ICMP echo requests received by b's stack: before=6 after=6 ---

--- tcpdump b:epair1b, bridge member (frames TO the virtual MAC 00:00:5e:00:01:0d) ---
10:45:21.508388 58:9c:fc:10:6e:52 > 00:00:5e:00:01:0d, ethertype 802.1Q (0x8100), length 102: vlan 3, p 0, ethertype IPv4 (0x0800), 172.16.2.1 > 172.16.2.13: ICMP echo request, id 33033, seq 0, length 64
10:45:22.577402 58:9c:fc:10:6e:52 > 00:00:5e:00:01:0d, ethertype 802.1Q (0x8100), length 102: vlan 3, p 0, ethertype IPv4 (0x0800), 172.16.2.1 > 172.16.2.13: ICMP echo request, id 33033, seq 1, length 64
10:45:23.605646 58:9c:fc:10:6e:52 > 00:00:5e:00:01:0d, ethertype 802.1Q (0x8100), length 102: vlan 3, p 0, ethertype IPv4 (0x0800), 172.16.2.1 > 172.16.2.13: ICMP echo request, id 33033, seq 2, length 64

--- tcpdump b:vlan3 (host 172.16.2.13) ---
10:45:21.508379 58:9c:fc:10:6e:52 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 172.16.2.13 tell 172.16.2.1, length 28
10:45:21.508382 26:00:ce:58:28:8e > 58:9c:fc:10:6e:52, ethertype ARP (0x0806), length 42: Reply 172.16.2.13 is-at 00:00:5e:00:01:0d, length 28
(nothing else - the echo requests never appear)

So ARP and CARP adverts go out via vlan3, but inbound unicast to 00:00:5e:00:01:0d stops between bridge0 and vlan3.
Do you want me to add this to github as well or shall we continue here?

[Patrick M. Hausen]

Reproduced in jails on a clean FreeBSD 15.1-RELEASE (GENERIC). No OPNsense code involved, so this is not specific to the 137513c backport — the native implementation behaves the same way, and 27.1 would inherit it.

Since there is a lot of ongoing work being put into the bridge subsystem, most of it by Lexi, did you open an issue on the FreeBSD bug tracker, or is there one, already?

[ArthurK.CICS]

Since there is a lot of ongoing work being put into the bridge subsystem, most of it by Lexi, did you open an issue on the FreeBSD bug tracker, or is there one, already?

No I have not and haven't checked either. Just ran the tests according to Monviech's instructions.

[Monviech (Cedrik)]

Very cool, thank you.

All three neighboring cases:

VLAN over bridge with normal interface MAC works.
CARP directly on the bridge works.
CARP on VLAN over a normal epair works.
-> Only CARP on VLAN over bridge fails.

This is definitely freebsd bug territory now as Patrick also suggested:
https://bugs.freebsd.org/bugzilla/

If you have opened one you can also backlink it here:
https://github.com/opnsense/src/issues

Thank you for your help!