VLANs spanning physical interfaces

[vldid]

 I need a few VLANs on multiple physical interfaces. There is nothing else to be on these interfaces.

What considerations should I put into arranging it?

Should I make one bridge across the interfaces and put VLANs on top of the bridge?

Or should I put VLANs on one interface and then make bridges between the VLANs and the rest of the physical interfaces?

I do not see a way to have VLANs on multiple interfaces in OPNsense docs.

Any other arrangement? Pros/cons?

Downstream will be WAPs and a managed L2 switch. Here is a rough diagram:



 I do not want to send VLAN 50 traffic to any port other than 1. Likewise, VLANs 10-40 traffic should not come to port 1. Port 0 is an uplink. Port 2 will either extend VLAN 50 access or have another dedicated VLAN in the future. All sorts of firewall configs should not be an issue and will be addressed later.

Adding another switch between OPNsense and WAPs is not a solution.

The project is a migration from Untangle - it works fine there, albeit convoluted. Repost from Reddit: https://www.reddit.com/r/opnsense/comments/1fz4tsx/vlans_spanning_physical_interfaces/.

[Patrick M. Hausen]

First: OPNsense/FreeBSD is not a switch.

That being said you can configure what you are seeking. The FreeBSD bridge code is reportedly good for 1G or more, but there are not yet any reports about 2.5G.

So you need to

- create a VLAN interface on each physical port - you probably got that far already
- create a bridge interface for each VLAN
- make all port/vlan interfaces a member of that bridge
- in the OPNsense context assign the respective bridge interface but not any of the members to e.g. LAN, OPT1, ...
- set the tunables according to the documentation, step 6

HTH,
Patrick

[vldid]

That is what I was looking for. Thank you very much!

[Greg_E]

Forget the bridge and connect the two WAP to the switch, save the headache of setting up the bridge.

[Patrick M. Hausen]

That's why the FreeBSD Foundation spent a ton of money on a complete overhaul and improvement of the FreeBSD bridge code so nobody uses it. 

[Greg_E]

I've seen a lot of users having trouble with bridges, but that was before the move in July. Does it really work now and work well? I haven't tried it but generally I would just go to the switch and run everything off of the switch because it seems more logical to me.

[Patrick M. Hausen]

Users are having problems because the documentation is somewhat arcane and there is no comprehensive guide apart from the - IMHO rather well written - simple LAN bridge guide in the OPNsense docs.

The overhaul took place in 2020 - people are just repeating old FUD over and over again ...

If configured correctly the FreeBSD bridge works as intended.

[cookiemonster]

That said, can someone point me to where to read the background to why we have been saying for freeBSD bridges, put the ip on the bridge, not on the members?
I know there is a reference to it on the freebsd handbook but I've not seen yet the why.

[Patrick M. Hausen]

That said, can someone point me to where to read the background to why we have been saying for freeBSD bridges, put the ip on the bridge, not on the members?
I know there is a reference to it on the freebsd handbook but I've not seen yet the why.

Having the IP address on a member interface breaks multicast for that broadcast domain. Something about scopes, the code path, etc. I did not research into the last detail. Mainly because I am not that deep in the actual code.

Confirmed in private conversation by Kristof Provost.

It's what makes most people show up at the forum in the TrueNAS context. TrueNAS auto-creates a bridge if you started a jail or VM. And then apps inside those containers that require multicast like Plex and other video players and friends "don't work".

And, oh, IPv6 won't work at all with the address on a member interface.

It's just how the FreeBSD stack works when you throw a bridge in at some point. Nothing wrong with it. The member interfaces become layer 2 ports. Like a layer 3 switch. You place the IP address on "interface Vlan 5", not on "interface GigabitEthernet 0/1".

HTH,
Patrick

[cookiemonster]

Thanks Patrick, most useful.
Indeed I have a host where I've been using Bastille to get some VNET jails going.
All is good for them and the IPs are still currently on the members, not the bridge. The members of the bridge are the a side of the epairs + host interface.
The b sides in the jails have IPs for each jail.
I have in my to-do to align it to the advice but have always been curious to the reason.
Thanks again.

[Patrick M. Hausen]

In the case of epair interfaces they have two ends. One on the host, which is probably a member of some bridge. And one end inside the jail. They are a virtual Ethernet cable.

The IP address is configured on the end inside the jail.

If you type

Code: [Select]

ifconfigon the host, you will find that all of the vnetX.Y interfaces indeed do not have an IP address. So everything is fine.

Possibly apart from your hosts physical network interface. If there is an IP address on that one, you should move it to the bridge. Otherwise IPv6 for your host and your jails is not going to work.

[vldid]

Forget the bridge and connect the two WAPs to the switch, save the headache of setting up the bridge.

As mentioned in the original post, an additional switch is not an option. The shelf in that space won't fit even a small switch, and powering it up will require outlet shenanigans.

It is also not feasible if you mean to connect the WAPs to the existing switch. The switch and WAPs are in different areas of the space, so pulling the wires back and forth between them through or around concrete walls is a non-starter. That also would not address a potential expansion of VLAN 50, which will have similar limitations.

Even if I had space for a second switch close to the router, I would give it a second thought. Yes, ASICs in a switch would make it more efficient. But, really, would they do so significantly?

I already have this Protectly. It draws what it draws. I can't imagine low-load bridge processing would add as much power draw as a separate switch with its inefficient power supply. And I only deal with one configuration—it is much more of an appliance (which I value) than two complexly connected devices from different vendors with their configurations to track and backup. Yes, it may stutter or heat up under an occasional load. That is fine a couple of times a year. Yes, it is likely not to be that fast. It won't be noticed in this application.

.........

Patrick, thank you very much, all of the above is very helpful!

[cookiemonster]

In the case of epair interfaces they have two ends. One on the host, which is probably a member of some bridge. And one end inside the jail. They are a virtual Ethernet cable.

The IP address is configured on the end inside the jail.

If you type

Code: [Select]

ifconfigon the host, you will find that all of the vnetX.Y interfaces indeed do not have an IP address. So everything is fine.

Possibly apart from your hosts physical network interface. If there is an IP address on that one, you should move it to the bridge. Otherwise IPv6 for your host and your jails is not going to work.

Yep, that's exactly how it is. Just the ip in the host side is on the interface not the bridge it is member of.
Luckily I'm not going anywhere near IPv6 until hopefully never.
Everything works fine but I intend on moving the ip to the bridge just for correctness and prevent some problem later.
Thanks and apologies OP for the noise on your thread.

[EricPerl]

Forget the bridge and connect the two WAPs to the switch, save the headache of setting up the bridge.

...

Even if I had space for a second switch close to the router, I would give it a second thought. Yes, ASICs in a switch would make it more efficient. But, really, would they do so significantly?

...

You would *NOT* give it a second thought?
I would have exactly the opposite reaction.
The power consumption wouldn't matter to me. On that front, I'd be more concerned about additional heat due to additional load (especially given I use a fanless mini-PC). It may not matter if you don't have servers on premises because client to client traffic would be minimal.

There's the complexity of creating/managing the OPNsense bridge (you wouldn't have started this thread).
A physical switch would have a super simple configuration. All traffic is tagged...

More importantly to me, it's cleaner from a layering perspective (switching in switches, routing at the router...).
And again, what you do on your network matters. e.g. access to servers or printers when the router is down.

[Patrick M. Hausen]

Folks, to each their own.

I am just (a bit) fighting the conception that the FreeBSD bridge is utterly worthless and you should buy a switch in any case.

I absolutely support the idea that e.g. a 4 port Protectli appliance should work in the same way as any consumer router - 1 port WAN, 3 ports LAN. And it does. Really. No problem at all if configured as documented.

I run 3 data centres with about 100 hosts total and over 1000 customer jails (a FreeBSD native container) and it's all bridged networking, sometimes over VLANs. If that was in any way unreliable, I would be out of business.