VLANs over a Bridge interface

[pankaj]

Hi,

I have 5 cheap L2 Trendnet switches (https://www.trendnet.com/products/edgesmart-switch/8-port-gigabit-EdgeSmart-switch-TEG-S80ES) that I would like to deploy on home network behind a Qotom device (6 ports) running OPNSense. First I tried the topology shown in diagram-01 below but only switch-A worked in this case. The VLANs on switch-B could neither access VLANs on switch-A nor OPNSense.

As an alternate, I'm thinking of combining the unused ports on OPNSense device to create a bridge and create VLANs on the bridge and connect each port of OPNSense to corresponding VLAN port on switch-A. And use port-1 on both switches to connect each other.

I'd need to take down my network to even try this one, so wanted to check in this forum if anyone has faced the same problem as I did in diagram-1 and if the alternate topology (diagram-2) will work in theory.

Thanks.

[Demusman]

Don't use the bridge, you'll regret it.
Post pictures of your switch vlan config.

[Patrick M. Hausen]

You can use a bridge on top of a VLAN, e.g. to bridge a VLAN interface and a physical port. You cannot use VLANs over a bridge interface.

[lecris]

Are there any updates on making a vlan on the bridge?

Currently I have setup a bridge with 2 vlan interfaces that I want to allow to freely communicate with each other on the same vlan, and allow routing to external network via the opnsense. To get this working I have one vlan interface with ip while the other is empty.

Problem is that it often gets connectivity issues to the opnsense ip and back, while it can (mostly) communicate well between the vlan interfaces.

I would like to keep some form of gating between the two interfaces so that I do not allow all vlans to pass through the other switch, so simply connecting the two switches is not a viable solution here. Ideally I would want that the traffic on this bridge to be untagged when it goes from vlan port to the bridge and re-tagged when it goes through the vlan.

[Patrick M. Hausen]

Are there any updates on making a vlan on the bridge?

This is fundamentally impossible in the FreeBSD network architecture and not going to change. The bridge interface does not support tagged frames.

Currently I have setup a bridge with 2 vlan interfaces that I want to allow to freely communicate with each other on the same vlan, and allow routing to external network via the opnsense. To get this working I have one vlan interface with ip while the other is empty.

Problem is that it often gets connectivity issues to the opnsense ip and back, while it can (mostly) communicate well between the vlan interfaces.

You need to assign the IP address to the bridge interface and leave both VLAN interfaces without an IP address.
This is also a fundamental property of FreeBSD and documented in the FreeBSD handbook. A bridge member interface MUST NOT have an IP address.

HTH,
Patrick

[lecris]

You need to assign the IP address to the bridge interface and leave both VLAN interfaces without an IP address.
This is also a fundamental property of FreeBSD and documented in the FreeBSD handbook. A bridge member interface MUST NOT have an IP address.

Setting ip address to the bridge interface is indeed the most intuitive, but if we do that the vlans cannot communicate with that. And what I am trying to get at is if there is anyway to fix that. Is that a bug? Why is that setup not feasible. Can't we get the network to be untagged from one port and tagged on the other?

[Patrick M. Hausen]

Create a VLAN interface. Create a bridge interface with that VLAN and another physical port as members. Voila, one port tagged, one port untagged.

In the OPNsense context you also need to change two tunables as in the documentation if you use a bridge. If you did not do that, that's a probable reason why your VLANs cannot communicate.

[lecris]

Probably we are miscommunication with each other here. Currently I know that this configuration works:
- eno1.vlan10 -> bridge
- eno2.vlan10 -> bridge
- no router ip, just as a switch (with appropriate firewall setting)

I suspect from your discussion that the following works too, but haven't confirmed:
- eno1.vlan10 -> bridge
- eno2.vlan20 -> bridge
- eno3 (untagged) -> bridge
- no router ip, just as a switch

But the issue that I encounter is in the following setup:
- eno1.vlan10 -> bridge
- eno2.vlan10 -> bridge
- router ip 192.168.1.1/24 on bridge interface
- communication between 192.168.1.2/24 on eno1.vlan10 and 192.168.1.3/24 on eno2.vlan10 works just fine
- communication between 192.168.1.2/24 on eno1.vlan10 and 192.168.1.1/24 on bridge does not work

But you are implying that it should work without a problem, but maybe there are tunables not set. If you are referring to the steps in https://docs.opnsense.org/manual/how-tos/lan_bridge.html#step-six, I have already tried that with no success. My suspicion was that it couldn't communicate because 192.168.1.1/24 was not on the same vlan 10, but you are saying that all traffic coming into eno1.vlan10 gets untagged when it is passed to bridge right? In that case I don't understand why my setup there doesn't work

[Patrick M. Hausen]

The layer 2 configuration is correct. Also putting an IP address on the bridge interface is correct. So you assigned the bridge interface to what exactly in Interfaces > Assignments? What are the firewall rules for that interface?

As soon as layer 3 is involved, of course the firewall kicks in.

And yes, you should definitely perform step 6.

[lecris]

To confirm that I am not going crazy, here are screenshots of what you said would be an appropriate configuration and how I can confirm there is no firewall issues.

As you can see there is no connection between the host on the vlan interface and bridge interface static ip. Outside connection to that IP works fine, but from vlan to bridge does not work. vlan to vlan worked last time I checked, but I need to fix my configuration to confirm that again.

[Patrick M. Hausen]

Is this virtualized? If yes, I'm out, sorry. Check promiscuous mode and MAC address spoofing on the hypervisor host. If this is not virtualized, what *is* that last screenshot? Never seen anything like this on OPNsense ...

[lecris]

No it is not virtualized. Last screen is a linux terminal of a host on the vlan, not sure what is cryptic about it. It shows I have the correct tag setup and ip within the subnet, but it cannot ping to the gateway. I have more screenshots, but there is a limit of 4 per post, so here are more.

Promiscuous mode was off on all, and I tried turning them all on, but still nothing. I don't know how to do mac spoofing, so any hints on that?

[Patrick M. Hausen]

The MAC spoofing and promiscuous mode are only relevant for a hypervisor host.

So the OPNsense is connected via that VLAN interface and a trunk (i.e. tagged) port to a switch and that Linux host is also connected to another trunk (tagged) port on that switch?

You are correct that the frames arrive at the bridge interface untagged when they come in via the VLAN interface. And when they leave via a different VLAN interface they are tagged again.

I'm a bit at the end of suggestions, now, without access to the machine. I'd grab tcpdump and watch what happens on the wire, now.

[lecris]

Could you or any other dev confirm that with the latest opnsense, the vlan is not broken?

Indeed that is the case. Is there another tool for more simple icmp requests?

Two curious things, for some reason I am able to get dhcp address on the machine, and when I do some actions like adding the vlan bridge to an untagged bridged or reverse, then for a brief millisecond the ping goes through in both ways

[Patrick M. Hausen]

There were some interfaces with broken VLAN support in FreeBSD. Try ifconfig <phys-interface-with-VLAN-on-top> promisc.