Bridge with VLAN devices behaves weird

[KoS]

Hi

I have an Alix APU with 3 ports.

• One port is for WAN
• One port is connected to a Switch where I have multiple VLANs, but where I can also have untagged traffic on it.
• One port should be bridge into one of the VLANs

To make the setup more flexible, I have bridged all VLANs and am assiging IP adress only to the bridge interfaces.

So the current configuration looks like this:

Code: [Select]

igb0
igb1 -> LAN
 igb1_untag
 igb1_vlan2
 igb1_vlan3
 igb1_vlan4
 igb1_vlan5
 igb1_vlan6
igb2 -> WAN

bridge0 = OpenVPN_Server_1, igb0, igb1_vlan3
bridge1 = igb1_vlan4
bridge2 = igb1_vlan2
bridge3 = igb1_vlan5
bridge4 = igb1_vlan6
bridge5 = OpenVPN_Server_2, igb1_untag

I have a DHCP server on each of the bridge interfaces. If I connect a device at the switch on a port of e.g. VLAN2, I can successfully receive an IP address via DHCP. But neither can I ping the router, nor do I see any traffic coming in on that bridge interface (tcpdump). Neither can I get any traffic out from the router on that bridge. I have checked the firewall rules, but as don't even see that the packets would get blocked, it seems the problem must be somewhere else. Is my setup with the bridge & vlans wrong? Shall I do it somehow else to get to my desired result? Any idea where I shall start debugging?

If I connect a device on an "untagged" port of the switch, I end up successfully on bridge5 and can access the router & the internet.

FYI, the OpenVPN_Server_1 and 2 are in TAP mode, as I need to have the full traffic (including broadcast) via the VPN.

[KoS]

I found the root cause of my problem: There is a limitation in OPNsense/FreeBSD that you cannot use a physical network interface with VLAN interfaces AND an untagged interface in bridges.
As I had the similar setup previously running on Linux, I didn't expect this to be a problem/limitation.

see e.g. here: https://redmine.pfsense.org/issues/11139

FYI: I have all "management" traffic un-tagged on the switches and all "data" traffic in different VLANs. e.g.  Ubiquiti UniFI access points have the "management" traffic always untagged and cannot be forced to use another VLAN. -> Even if it would be possible to change the management traffic to a tagged VLAN, it won't be possible to just plug-in a new access point out-of-the-box and it configures itself automatically by connecting to the UniFi controller, as you would first need to configure it manually.

[Mark Rose]

Thank you for this!

I've spent countless hours trying to figure out why my VLANs had no access.

I'm also running Unifi APs. I also have an untagged bridge and several tagged bridges over the same interfaces.

It would be nice if the OPNsense interface at least warned that this configuration is unsupported.

[Patrick M. Hausen]

@KoS can't you run the management VLAN tagged on the trunk port to OPNsense and untagged for all other ports, specifically the ones connected to your APs?

On all switches I know the so called "native VLAN" is a per port setting.

[KoS]

@pmhausen
sure this is possible and is what I have to do for new installations where I want to use OPNsense.
This makes the setup less transparent, as not all trunk ports on the switch can be configured the same way. in the end it is ONLY the trunk port for OPNsense that needs to be configured differently, as the trunk ports for uplinks to other switches or APs can be configured all the same way.  -> and on existing installations i cannot just replace the existing router box (running voyage linux on the Alix APU boards) as I first need to re-configure the port on the switch.

[Patrick M. Hausen]

I don't use the native VLAN anywhere. I have a special dummy VLAN (1001) assigned to all trunk ports. And honestly I do not fully understand the problem with new devices and VLAN 1 ...

I also use VLAN 1 for management. But all ports are configured as access ports by default. So plug in new device - connectivity established. Then if the new device is a network "thing" needing trunking, change switchport to trunk. If the device is a customer server I need to change the VLAN before plugging in, anyway.

Are you configuring your switches to enable trunking dynamically? That's what I would never do. All ports are configured statically as access or trunk as necessary.

But yes, you need to change your procedures, which nobody likes to do. I am fine with FreeBSD in that regard because I think a "native VLAN" should not exist in the standard at all ...

Kind regards,
Patrick