Messages

This is definitely freebsd bug territory now as Patrick also suggested:
https://bugs.freebsd.org/bugzilla/

If you have opened one you can also backlink it here:
https://github.com/opnsense/src/issues

Update on the upstream reporting: this was already in FreeBSD's bugzilla — PR 296106
(carp/vlan/bridge: CARP IP not responding on top of bridge),
filed in June against the native 15.x implementation (VLANFILTER bridge, untagged member).
Same signature we saw: interface address on the bridge-vlan works, CARP VIP dead, frames
visible on the bridge but never on the vlan interface.

So instead of filing a duplicate we added our findings there as a comment: the minimal
two-jail repro (entire datapath in-kernel — epairs + vnet jails, no NIC or hypervisor
vswitch touches a frame, which rules out the ESXi theory from earlier in that PR), plus the
code-level cause from reading if_bridge.c: bridge_input()'s GRAB_OUR_PACKETS only consults
the bridge's own if_carp (CARP_CHECK_WE_ARE_DST(bifp)), never the vlan child's, and vlan
demux happens only after a frame is accepted as local — so the vlan child's virtual MAC
always fails the "ours" test and gets flooded instead of delivered.

Notable from that PR: comment #9 confirms it independently and describes an epair-based
workaround (CARP on the b-side of an epair whose a-side is a bridge member, so the virtual
MAC actually gets learned). Hacky, but it exists.

Since there is a lot of ongoing work being put into the bridge subsystem, most of it by Lexi, did you open an issue on the FreeBSD bug tracker, or is there one, already?

No I have not and haven't checked either. Just ran the tests according to Monviech's instructions.

[carp_jail_setup.sh]

Reproduced in jails on a clean FreeBSD 15.1-RELEASE (GENERIC). No OPNsense code involved, so this is not specific to the 137513c backport — the native implementation behaves the same way, and 27.1 would inherit it.

I extended your setup script with CARP. Jail b gets a VIP on its vlan3 (over the bridge) and a second VIP untagged on bridge0 itself; jail c gets a VIP on its vlan3 (directly on the epair) as control. Fresh 15.1 VM, kldload carp, nothing else.

Pings from jail a:

• interface addresses (172.16.2.2, 172.16.3.2, 172.16.2.3): all fine
• VIP on vlan-over-epair, jail c (172.16.2.23): fine
• VIP untagged on bridge0 itself, jail b (172.16.3.13): fine
• VIP on vlan3 over bridge0, jail b (172.16.2.13): ARP resolves, then 100% loss. The echo requests show up on epair1b and on bridge0, never on vlan3, and netstat's ICMP input counter in jail b stays flat.

carp_jail_setup.sh (your setup + CARP)

Code Select Expand

#!/bin/sh

# Extension of Monviech's baseline jail scripts (opnsense/core PR #10584)

# to reproduce: CARP VIP on a VLAN-over-bridge interface does not receive unicast.

#

# jail a = client (tagged vlan3 + untagged leg)

# jail b = DUT: vlan3 stacked on an internal bridge0

#          -> CARP VIP on vlan3   (vhid 13, 172.16.2.13)  = the bug case

#          -> CARP VIP on bridge0 (vhid 33, 172.16.3.13)  = control, untagged on the bridge itself

# jail c = control: vlan3 directly on the epair

#          -> CARP VIP on vlan3   (vhid 23, 172.16.2.23)  = control, no bridge involved

kldload -n carp

ifconfig epair0 create
ifconfig epair1 create
ifconfig epair2 create

ifconfig bridge0 create \
    addm epair0a \
    addm epair1a \
    addm epair2a \
    up

ifconfig epair0a up
ifconfig epair1a up
ifconfig epair2a up

jail -c name=a path=/ persist vnet allow.raw_sockets=1
jail -c name=b path=/ persist vnet allow.raw_sockets=1
jail -c name=c path=/ persist vnet allow.raw_sockets=1

ifconfig epair0b vnet a
ifconfig epair1b vnet b
ifconfig epair2b vnet c

# jail a - client

jexec a ifconfig epair0b inet 172.16.3.1/24 up
jexec a ifconfig vlan3 create vlan 3 vlandev epair0b
jexec a ifconfig vlan3 inet 172.16.2.1/24 up

# jail b - DUT: vlan on bridge, CARP on both layers

jexec b ifconfig epair1b up
jexec b ifconfig bridge0 create addm epair1b up
jexec b ifconfig bridge0 inet 172.16.3.2/24
jexec b ifconfig vlan3 create vlan 3 vlandev bridge0
jexec b ifconfig vlan3 inet 172.16.2.2/24 up
jexec b ifconfig vlan3 vhid 13 pass testpass alias 172.16.2.13/32
jexec b ifconfig bridge0 vhid 33 pass testpass alias 172.16.3.13/32

# jail c - control: vlan directly on the epair, CARP on it

jexec c ifconfig epair2b up
jexec c ifconfig vlan3 create vlan 3 vlandev epair2b
jexec c ifconfig vlan3 inet 172.16.2.3/24 up
jexec c ifconfig vlan3 vhid 23 pass testpass alias 172.16.2.23/32

carp_jail_test.sh

Code Select Expand

#!/bin/sh

# Test companion to carp_jail_setup.sh - CARP VIP on VLAN-over-bridge repro.

# Expected on an affected kernel:

#   interface addresses        : all reachable

#   VIP on vlan-over-epair (c) : reachable            (control 1)

#   VIP untagged on bridge (b) : reachable            (control 2)

#   VIP on vlan-over-bridge (b): ARP resolves, 100% loss - frames visible on

#                                the bridge member and bridge0, never on vlan3,

#                                ICMP input counter on b does not move.  (BUG)

LOG=${1:-carp-vlan-bridge-test.log}
TMP=/tmp/carp-vlan-bridge-test.$$

wait_master() { # jail iface
    n=0
    while [ $n -lt 15 ]; do
        jexec "$1" ifconfig "$2" | grep -q "carp: MASTER" && return 0
        sleep 1; n=$((n+1))
    done
    echo "WARNING: $1/$2 never reached MASTER"
    jexec "$1" ifconfig "$2"
}

icmp_in_echo() { # jail - echo requests received (Input histogram: echo)
    jexec "$1" netstat -s -p icmp | sed -n '/Input histogram/,/^[^[:space:]]/p' \
        | awk '/echo:/{print $2; exit}'
}

{
    echo "=== CARP election state ==="
    wait_master b vlan3
    wait_master b bridge0
    wait_master c vlan3
    jexec b ifconfig vlan3
    jexec b ifconfig bridge0
    jexec c ifconfig vlan3

    echo
    echo "=== baseline: interface addresses ==="
    jexec a ping -c 2 -t 4 172.16.2.2 || true
    jexec a ping -c 2 -t 4 172.16.3.2 || true
    jexec a ping -c 2 -t 4 172.16.2.3 || true
   
    echo
    echo "=== CONTROL 1: CARP VIP on vlan-over-EPAIR (c, 172.16.2.23, vhid 23) ==="
    jexec a ping -c 2 -t 4 172.16.2.23 || true
   
    echo
    echo "=== CONTROL 2: CARP VIP UNTAGGED on the bridge itself (b, 172.16.3.13, vhid 33) ==="
    jexec a ping -c 2 -t 4 172.16.3.13 || true
   
    echo
    echo "=== BUG CASE: CARP VIP on vlan-over-BRIDGE (b, 172.16.2.13, vhid 13) ==="
    BEFORE=$(icmp_in_echo b)
   
    jexec b tcpdump -nlei epair1b ether host 00:00:5e:00:01:0d >"${TMP}.member" 2>&1 &
    PID0=$!
    jexec b tcpdump -nlei bridge0 ether host 00:00:5e:00:01:0d >"${TMP}.bridge" 2>&1 &
    PID1=$!
    jexec b tcpdump -nlei vlan3 host 172.16.2.13 >"${TMP}.vlan" 2>&1 &
    PID2=$!
   
    sleep 1
    jexec a ping -c 3 -t 6 172.16.2.13 || true
    sleep 1
   
    kill "$PID0" "$PID1" "$PID2" 2>/dev/null
    wait "$PID0" "$PID1" "$PID2" 2>/dev/null
   
    AFTER=$(icmp_in_echo b)
   
    echo
    echo "--- client (a) neighbor table: virtual MAC resolved? ---"
    jexec a arp -an | grep 172.16.2.13 || echo "(no ARP entry)"
   
    echo
    echo "--- ICMP echo requests received by b's stack: before=$BEFORE after=$AFTER ---"
   
    echo
    echo "--- tcpdump b:epair1b, bridge member (frames TO the virtual MAC 00:00:5e:00:01:0d) ---"
    cat "${TMP}.member"
   
    echo
    echo "--- tcpdump b:bridge0 (frames TO the virtual MAC 00:00:5e:00:01:0d) ---"
    cat "${TMP}.bridge"
   
    echo
    echo "--- tcpdump b:vlan3 (host 172.16.2.13) ---"
    cat "${TMP}.vlan"

} >"$LOG" 2>&1

rm -f "${TMP}.member" "${TMP}.bridge" "${TMP}.vlan"
cat "$LOG"

carp_jail_teardown.sh

Code Select Expand

#!/bin/sh
for j in a b c; do
    jail -r "$j" 2>/dev/null
done
for i in epair0a epair1a epair2a; do
    ifconfig "$i" destroy 2>/dev/null
done
ifconfig bridge0 destroy 2>/dev/null

Log excerpts (full log available if useful):

Code Select Expand

$ uname -a
FreeBSD carp-jailtest 15.1-RELEASE FreeBSD 15.1-RELEASE releng/15.1-n283562-96841ea08dcf GENERIC amd64

=== CONTROL 1: CARP VIP on vlan-over-EPAIR (c, 172.16.2.23, vhid 23) ===
2 packets transmitted, 2 packets received, 0.0% packet loss

=== CONTROL 2: CARP VIP UNTAGGED on the bridge itself (b, 172.16.3.13, vhid 33) ===
2 packets transmitted, 2 packets received, 0.0% packet loss

=== BUG CASE: CARP VIP on vlan-over-BRIDGE (b, 172.16.2.13, vhid 13) ===
3 packets transmitted, 0 packets received, 100.0% packet loss

--- client (a) neighbor table: virtual MAC resolved? ---
? (172.16.2.13) at 00:00:5e:00:01:0d on vlan3 expires in 1193 seconds [vlan]

--- ICMP echo requests received by b's stack: before=6 after=6 ---

--- tcpdump b:epair1b, bridge member (frames TO the virtual MAC 00:00:5e:00:01:0d) ---
10:45:21.508388 58:9c:fc:10:6e:52 > 00:00:5e:00:01:0d, ethertype 802.1Q (0x8100), length 102: vlan 3, p 0, ethertype IPv4 (0x0800), 172.16.2.1 > 172.16.2.13: ICMP echo request, id 33033, seq 0, length 64
10:45:22.577402 58:9c:fc:10:6e:52 > 00:00:5e:00:01:0d, ethertype 802.1Q (0x8100), length 102: vlan 3, p 0, ethertype IPv4 (0x0800), 172.16.2.1 > 172.16.2.13: ICMP echo request, id 33033, seq 1, length 64
10:45:23.605646 58:9c:fc:10:6e:52 > 00:00:5e:00:01:0d, ethertype 802.1Q (0x8100), length 102: vlan 3, p 0, ethertype IPv4 (0x0800), 172.16.2.1 > 172.16.2.13: ICMP echo request, id 33033, seq 2, length 64

--- tcpdump b:vlan3 (host 172.16.2.13) ---
10:45:21.508379 58:9c:fc:10:6e:52 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 172.16.2.13 tell 172.16.2.1, length 28
10:45:21.508382 26:00:ce:58:28:8e > 58:9c:fc:10:6e:52, ethertype ARP (0x0806), length 42: Reply 172.16.2.13 is-at 00:00:5e:00:01:0d, length 28
(nothing else - the echo requests never appear)

So ARP and CARP adverts go out via vlan3, but inbound unicast to 00:00:5e:00:01:0d stops between bridge0 and vlan3.
Do you want me to add this to github as well or shall we continue here?

[Bug report — CARP VIP on VLAN-over-bridge]

Bug report — CARP VIP on VLAN-over-bridge

We are running 137513c on 26.7.1_1 per the instructions earlier in this thread — the feature itself works great for us (bridged SFP+ fabric in a virtualized HA pair). With CARP on top of the VLAN-on-bridge interfaces, however, we hit what looks like a gap in if_bridge's local-delivery path:

ARP and CARP advertisements work, but unicast to the VIP's virtual MAC is never delivered.

Setup

• bridge0 over five physical members (VM with PCIe passthrough NICs, but nothing here is virtualization-specific).
• L3 untagged directly on bridge0 (assigned interface), plus tagged VLANs vlan0.10 ... vlan0.19 with parent bridge0 (per the patch), each assigned with a static IPv4 and a CARP VIP (vhid = VLAN tag).
• Second node with an identical layout; CARP advertisements cross a bridged peer link.

What works

• L3 on the VLAN-on-bridge interfaces: unicast to the interface's own address/MAC is fine.
• CARP election per VLAN: advertisements are sent and received tagged, MASTER/BACKUP negotiate correctly, failover and preempt work.
• ARP for the VIP: clients resolve it and learn the virtual MAC (00:00:5e:00:01:<vhid>).
• A CARP VIP configured on bridge0 itself (the untagged side): fully functional including unicast — this one is not affected.

The bug

Unicast frames addressed to the CARP virtual MAC of a VLAN-on-bridge interface are never delivered to that interface's IP stack. Observed on the box itself:

Code Select Expand


# tcpdump -i bridge0 -e -nn ether host 00:00:5e:00:01:0a

xx:xx:xx:xx:xx:xx > 00:00:5e:00:01:0a, 802.1Q, vlan 10, IPv4,
    10.0.10.53 > 10.0.10.3: ICMP echo request ...
(frames arrive continuously)

# tcpdump -i vlan0.10 -nn host 10.0.10.3

(nothing - ever)

So the frames reach bridge0 with the correct tag, but the bridge treats unicast to the VLAN child's virtual MAC as unknown unicast (floods it to the members) instead of delivering it upward. The second HA node shows the same split from its own vantage point: it can ping the untagged VIP on bridge0, but none of the tagged VIPs — with identical CARP config (unique vhids, verified) and a healthy MASTER/BACKUP election on every vhid. The client side shows the classic signature: the VIP's ARP resolves, the neighbor entry then hangs in PROBE, pings to the VIP are 100% loss while pings to the interface address on the same VLAN work — and the firewall log shows the flows as passed, so everything looks healthy.

Suspected cause

The bridge's "is this ours?" check on the RX path appears to match member MACs, the bridge's own MAC, and CARP MACs bound to the bridge interface itself — but not CARP vhids attached to VLAN interfaces stacked on the bridge. The untagged case working while every tagged case fails is consistent with that.

Repro

• Apply 137513c on 26.7.1, create bridge0 with at least one member, assign and address it.
• Create vlan0.10 with parent bridge0, assign, address it (e.g. 10.0.10.1/24).
• Add a CARP VIP 10.0.10.3/24, vhid 10, on that VLAN interface.
• From any host on VLAN 10: ping 10.0.10.1 → works; ping 10.0.10.3 → ARP resolves, then 100% loss. tcpdump as above shows the frames dying inside the bridge.

Impact

CARP VIPs on VLANs-over-bridge cannot serve as gateway addresses at all, which silently breaks HA-gateway designs on this topology. It is doubly deceptive because everything observable (election, ARP, firewall logs) looks fine.

Is this addressed in the native FreeBSD 15 / 27.1 implementation, or does it need a separate if_bridge fix (consulting the CARP MACs of stacked VLAN devices in the local-delivery decision)? Happy to test patches — we can reproduce this on demand.

Tests welcome :)

I did today and can confirm it's working on a small MS01.
I will be testing on a Dell R740 in the coming week too.
Our usecase is that we have a 4 port sfp+ NIC passed through to an OPNSense VM on a Dell that is part of a three node Proxmox Cluster.
The vlan-on-bridge feature is actually exactly what we need right now: Bridging the 3 ports for the Proxmox hosts and plugging them straight into the virtualized OPNSense, because we now mitigate the need for an additional switch with this while getting the Proxmox VLAN aware.
Thanks for adding this.
Any idea which update it will land in or when it will be officially rolled out?

Will report back when I have tested with the passed through NIC on the Dell.

Hey everyone! :)

I have searched for the problem and tried around for two days now but I can't seem to get rid of a problem, which is why I wanted to ask for help here.

I set up a Wireguard outbound connection to ProtonVPN three days ago following the Wireguard ProtonVPN Road Warrior guide (https://docs.opnsense.org/manual/how-tos/wireguard-client-proton.html).
I got it working and it does as expected.
During setting up something else I had to reboot my OPNsense and after the reboot I did not have connection anymore because the gateway showed as "down". Furthermore in my notfications I can see "loop detected for wg2" (for reference: I have two more inbound connections for remote devices set up, wg0 and wg1).

When I disable the gateway, I get connection again through my WAN gateway. If I let it sit for a while and re-enable the wg-gateway, it works and all my devices are routed out through the tunnel again.

The problem to me seems like that the outbound wireguard connection tries to establish its connection through the wg-gateway upon boot and goes into a loop. If i manually disable the gateway, wireguard connects to proton via the wan gateway and now that the connection stands, i can reenable the wg-gateway and the traffic is routed through.
Now... is there a way to manually delay the start of the wg-outbound gateway upon boot? Or did I set something up incorrectly? Does anyone have an idea how I can get the wg-outbound instance to connect via the wan gateway and not go into a loop? I do want the other instances (inbound, wg0 and wg1) to be routed out through wg2 though, which makes it a bit more complicated I think.

Anyone got any ideas how I can fix this?
Appreciate any help! :)

Maybe that sentence was a little confusing, sorry. Let me clear it up:

S1 (Zyxel XGS1250) is connected to S2 (which sits in a different room, TP-Link TL-SG108E) via cables in the wall. If I plug in my laptop via that wall plug, my traffic goes through the wall to S1, which then sends the packets to S3 (MikroTik hAP ax²) instead of upstream to the Sense. If I plug in S2 and connect devices to it, then traffic gets send through S2 to S1 and from there on out.

My network is as follows:
I have my LAN which is 10.0.0.0/24. In the LAN there's only the switches and nothing else.
I have a server vLAN (10) which holds my NAS and my Proxmox Server. They are in the 10.0.10.0/24 range.
I have another "services" vLAN (20), which holds all the services that run on the Proxmox Server (like the PiHole etc) in the 10.0.20.0/24 range.
Then there's a work vlan (200) at 10.0.200.0/24 and an IoT vlan (90) at 10.0.90.0/24 as well as two networks for traffic coming in through wireguard at 10.0.100.0/24 and 10.0.101.0/24.

The network is therefor quite separated and I should not have overlapping subnets - at least to my knowledge and understanding.

What is supposed to happen is that S1 shows up as connected in the OpnSense, which is doesn't. It is also supposed to be reachable from that side (via the wireguarded mobile devices that reach the Sense). It is highly confusing that the whole network is reachable *through* the switch but the switch itself is only accessible from behind the switch and not from in front.
Testing was performed by connected multiple devices to the network behind the switch and connecting to it, which is possible from everywhere behind it.

In short: Everything works but communication between the switch and the Sense. Everything communicated *through* the switch with the sense works, everything communicating behind the switch with eachother or the switch works as well.
There's simply no connection via Sense and Switch directly when it's assigned a static IP of 10.0.0.10. When it's assigned a dynamic IP (for example 10.0.0.13 handed out by the Sense) it shows up as connected in the Sense. When it is handed the static IP of 10.0.0.10, the Switch actually receives the IP and reacts to request on that IP but shows as disconnected on the Sense.

I have to also note that this very setup has worked flawlessly before. For some reason the connection between switch and Sense just broke and I can't figure out why, especially because everything else gets passed through accordingly.

Hello everyone.  :)

Just set up my account after reading the forums for over a year now (thanks for all the help so far that I got from reading here!). I have moved in to a new flat and set up a new MS-01 as my OPNSense. Everything worked so far but I have an extremely weird DHCP behavior that I can't seem to figure out and so I wanted to ask for help:

The most important thing first:
I upgraded this morning to OPNsense 24.7.4, as I thought it might be a bug that is already resolved but this didn't help. Back to the setup:

Behind my Sense sits my Main Switch, a Zyxel ("S1" @10.0.0.10). Attached to the switch I have a NAS and a Proxmox server as well as two other switches, a TP-Link in my office ("S2" @ 10.0.0.20) and a Mikrotik in the living room that also acts as an access point ("S3" @ 10.0.0.30).
When I am in my network, for example connected via my workstation at 10.0.200.11 (WorkVLAN 200, connected to S2) I can reach the Zyxel at 10.0.0.10. I can also reach the Sense from there and reach out to the internet.
However, on my Sense GUI S1 shows as offline. When I wireguard into my Sense with my phone I can also *not* see S1 when I browse to 10.0.0.10. When I am in my homenetwork with my phone via S3 Access Point however I can see it. That means the whole network that sits behind S1 can see S1 but everything in front of it (just the OPNSense and the remote devices that wireguard in) cannot.

Now.. this is where it gets confusing:
When I wireguard in from outside, I can however reach S2 and S3 that sit *behind* S1. I can also reach all my services on the Proxmox server. S2, S3, Proxmox and NAS, all connected to S1 show up as "online" in the DHCP leases.

So far so confusing.. now let's add some more confusion:
I figured something must be wrong with the switch so I reset it. No change. So I removed the static DHCP lease for S1 and gave it a dynamic IP. It now showed up at 10.0.0.13 as online but I could not reach it from anywhere, neither from Wireguarded phones on the Sense nor from the Workstation behind S2. When reassigning a static lease for 10.0.0.10 again, it showed as offline, *however* it did receive this IP address and was now accessible again from behind it but not from in front.

I played that through several times with static and dynamic leases, rebooted the Sense and S1 many times, reset S1 countless times but nothing changes.

It has all worked in the past and I just set up Wireguard to Proton so I first thought that it might have something to do with the Sense routing traffic that is destined for S1 through the tunnel to Proton instead of through the SFP+ port connected to S1. On closer thought this doesn't make too much sense either though, because S2 and S3 that sit behind the same SFP+ trunk are indeed reachable. I also just recently added S3 and the access point and when I plug in my laptop to the wall where S2 is usually connected to, S1 routes me to S3 instead of out to the Sense. So it might also have something to do with my VLAN setup but that is already quite the far fetch.

Furthermore I have a lot of errors in my DHCP log that look like this for pretty much every static IP that I have assigned:

Code Select Expand


2024-09-15T09:09:52 Error dhcpd from the dynamic address pool for 10.0.20.0/24
2024-09-15T09:09:52 Error dhcpd Remove host declaration s_opt3_1 or remove 10.0.20.20
2024-09-15T09:09:52 Error dhcpd Dynamic and static leases present for 10.0.20.20.
2024-09-15T09:01:58 Error dhcpd from the dynamic address pool for 10.0.20.0/24
2024-09-15T09:01:58 Error dhcpd Remove host declaration s_opt3_0 or remove 10.0.20.10
2024-09-15T09:01:58 Error dhcpd Dynamic and static leases present for 10.0.20.10.
2024-09-15T09:01:55 Error dhcpd from the dynamic address pool for 10.0.20.0/24



It is an extremely odd behavior and a bit above my league. Does anyone have an idea? I'd appreciate the help a lot.