OPNsense 26.7.4 - VLAN traffic failing over X550/LAGG/bridge after upgrade

[merrins63]

Hi all,

I'm troubleshooting a VLAN issue that appeared after upgrading to OPNsense 26.7.4 / FreeBSD 15.1.

My setup is roughly:
Cisco 2960X
    |
Po3 LACP / 802.1Q
    |
Intel X550 ix0 + ix1
    |
  lagg0
    |
 VLAN interfaces
    |
 OPNsense bridges
    |
Kea DHCP

I also have an Intel i226 2.5 GbE trunk, with corresponding VLAN interfaces bridged to the X550/LAGG VLAN interfaces.

This affects all VLANs traversing the Cisco 2960X /   Intel X550 LAGG path. VLAN 15 is simply the VLAN I am using for troubleshooting.

Clients on the Cisco side fail DHCP and remain on 169.254.x.x.

Cisco switch looks healthy: Po3 is UP, both LACP members are bundled, VLAN 15 is forwarding, and the test client MAC is correctly learned on its VLAN 15 access port.

The interesting part is simultaneous packet captures of the same DHCP transaction:

bridge1:      DHCP REQUEST ✓   ACK ✓
vlan0.1.15:   DHCP REQUEST ✓   ACK ✓
lagg0:        DHCP not visible
ix0 / ix1:    DHCP not visible

Kea therefore appears to receive the request and generate an ACK, but the client never receives/configures the lease.

I found FreeBSD bug 276936, describing packet-forwarding problems involving ixgbe, VLAN interfaces and bridges. The topology isn't identical, but it looks potentially relevant.

Has anyone experienced similar issues with OPNsense 26.7, Intel X550, LAGG/LACP, VLANs and bridges, or have suggestions for further diagnostics?

Any help or feedback would be greatly appreciated, as I am stuck until I can fix this, as my network relies on the VLAN Bridges to connect my home network

Thanks in advance

[merrins63]

Hi Team

I have found that there is a bug, meaning I have found the root cause of the issue

The issue is that when you use Bridged VLANs, you cannot use Kea DHCP, it simply doesnt allow you to connect

When you use Bridged VLANS, you must use ISC Legacy DHCP, I have confirmed my connection issue is resolved

If anyone in the OPNsense forum knows how to report a bug, as I believe this is a bug in the system

Regards

[Patrick M. Hausen]

Can you post the full output of "ifconfig", please? You can remove public IP addresses.

[merrins63]

Hi Patrick

Thanks for replying to my post!

Please see the ifconfig output below. I've redacted a few personal details from the VLAN/interface descriptions, but the underlying configuration and topology remain unchanged, so hopefully it provides everything needed to understand the setup.

My ultimate goal is to move away from Legacy ISC DHCP and transition fully to Kea DHCP. That was actually one of the objectives when I rebuilt the router from scratch on OPNsense 26.7.4.

At the moment, the bridged VLAN configuration is working correctly with Legacy ISC DHCP, whereas I have been unable to achieve the same result with Kea.

Thanks again for taking the time to look at this.
----------------------------------------------------
root@OPNsense:~ # ifconfig

ix0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,HWSTATS,MEXTPG>
ether (redacted MAC)
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4803828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,HWSTATS,MEXTPG>
ether (redacted MAC)
hwaddr (redacted MAC)
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

igc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: TRUNK (lan)
options=4902028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NETMAP,HWSTATS,MEXTPG>
ether (redacted MAC)
inet 10.100.14.1 netmask 0xfffffff8 broadcast 10.100.14.7
groups: Interface_GROUP
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

igc1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WAN (wan)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether (redacted MAC)
inet6 (redacted IPv6)
groups: Interface_GROUP
media: Ethernet autoselect
status: no carrier
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet 10.100.100.1 netmask 0xffffffff
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

enc0: flags=0 metric 0 mtu 1536
options=0
groups: enc
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

pflog0: flags=0 metric 0 mtu 33152
options=0
groups: pflog

pfsync0: flags=0 metric 0 mtu 1500
options=0
maxupd: 128 defer: off version: 1500
syncok: 1
groups: pfsync

lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAGG_INTERFACE (opt1)
options=4903828<VLAN_MTU,JUMBO_MTU,WOL_UCAST,WOL_MCAST,WOL_MAGIC,NETMAP,HWSTATS,MEXTPG>
ether (redacted MAC)
hwaddr 00:00:00:00:00:00
inet 10.100.23.1 netmask 0xfffffff8 broadcast 10.100.23.7
laggproto lacp lagghash l2,l3,l4
laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

vlan0.1.10: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: (redacted VLAN 10 interface)
options=4000000<MEXTPG>
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

vlan0.2.10: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: (redacted VLAN 10 interface)
options=4000000<MEXTPG>
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

vlan0.1.15: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: (redacted VLAN 15 interface)
options=4000000<MEXTPG>
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

vlan0.2.15: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: (redacted VLAN 15 interface)
options=4000000<MEXTPG>
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

vlan0.1.20:
description: (redacted VLAN 20 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
status: active

vlan0.2.20:
description: (redacted VLAN 20 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active

vlan0.1.30:
description: (redacted VLAN 30 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
status: active

vlan0.2.30:
description: (redacted VLAN 30 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active

vlan0.1.35:
description: (redacted VLAN 35 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 35 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
status: active

vlan0.2.35:
description: (redacted VLAN 35 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 35 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active

vlan0.1.40:
description: (redacted VLAN 40 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
status: active

vlan0.2.40:
description: (redacted VLAN 40 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active

vlan0.1.45:
description: (redacted VLAN 45 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 45 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
status: active

vlan0.2.45:
description: (redacted VLAN 45 interface)
ether (redacted MAC)
groups: vlan (redacted VLAN groups)
vlan: 45 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active

bridge0:
description: BRIDGE_VLAN10_INTERFACE
ether (redacted MAC)
inet 10.100.10.1 netmask 0xffffff00 broadcast 10.100.10.255
proto rstp maxaddr 2000 timeout 1200
member: vlan0.2.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        port 11 priority 128 path cost 20000 vlan protocol 802.1q
member: vlan0.1.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        port 10 priority 128 path cost 55 vlan protocol 802.1q
groups: bridge (redacted bridge groups)

bridge1:
description: BRIDGE_VLAN15_INTERFACE
ether (redacted MAC)
inet 10.100.15.1 netmask 0xffffff00 broadcast 10.100.15.255
proto rstp maxaddr 2000 timeout 1200
member: vlan0.2.15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        port 13 priority 128 path cost 20000 vlan protocol 802.1q
member: vlan0.1.15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        port 12 priority 128 path cost 55 vlan protocol 802.1q
groups: bridge (redacted bridge groups)

bridge2:
description: BRIDGE_VLAN20_INTERFACE
ether (redacted MAC)
inet 192.168.50.1 netmask 0xffffff00 broadcast 192.168.50.255
proto rstp maxaddr 2000 timeout 1200
member: vlan0.2.20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
member: vlan0.1.20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
groups: bridge (redacted bridge groups)

bridge3:
description: BRIDGE_VLAN30_INTERFACE
ether (redacted MAC)
inet 10.100.30.1 netmask 0xffffff00 broadcast 10.100.30.255
member: vlan0.2.30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
member: vlan0.1.30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
groups: bridge (redacted bridge groups)

bridge4:
description: BRIDGE_VLAN35_INTERFACE
ether (redacted MAC)
inet 10.100.35.1 netmask 0xffffff00 broadcast 10.100.35.255
member: vlan0.2.35 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
member: vlan0.1.35 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
groups: bridge (redacted personal bridge groups)

bridge5:
description: BRIDGE_VLAN40_INTERFACE
ether (redacted MAC)
inet 10.100.40.1 netmask 0xffffff00 broadcast 10.100.40.255
member: vlan0.2.40 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
member: vlan0.1.40 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
groups: bridge (redacted personal bridge groups)

bridge6:
description: BRIDGE_VLAN45_INTERFACE
ether (redacted MAC)
inet 10.100.45.1 netmask 0xffffff00 broadcast 10.100.45.255
member: vlan0.2.45 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
member: vlan0.1.45 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
groups: bridge (redacted personal bridge groups)

zen0:
flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
groups: tun (redacted VPN group)
---------------------------------------------------------------------

[Patrick M. Hausen]

Alright. You VLAN and bridge configuration is correct. You assigned the OPNsense interfaces to the bridges and there are no IP addresses on any bridge members.

I assume you enabled Kea on the bridge interfaces, too? Can't do anything but use the assigned symbolic names in the UI if I am not mistaken. Are you trying with raw or UDP sockets? Should be raw, IMHO.

Kind regards,
Patrick