is this going to work? (Bridges with UniFi hardware)

[hoondi]

Hello,


Am new to OPNSense (and also UniFi hardware) and have been learning a lot when it comes to bridges, vlan networks, physical networks and vlans bridged with physical networks.



Anyway, it has been fun to tinker, reset, tinker again, reset again and have now come to the point of asking for advice before I continue.
I've attached a PNG of my ideal setup and I'm getting stuck on the management side of things.


What I'm seeing is that if I include the physical ports in the MGMT bridge, the U6Pro WAPs can no longer issue respective IP addresses for any of the vlans/SSIDs. Clients connect to the SSID, give up and self assign. Attempting to statically assign an IP address does not work either and so there is no route/plumbing for any of the vlans back to the OPNSense router.


What I don't understand though, is that I'm able to ping both WAPS (192.168.1.249 & 192.168.1.251) from the Mac Mini, they just don't seem to parse the VLANs onto any WiFi clients. Any endpoint directly patched into the router (TV, MacMini, AppleTV etc etc) all work fine, and so this tells me that the vlans are not able to extend past/through the flex mini switches for some reason?


If I remove the physical ports from the MGMT vlan, everything works!
WiFi devices receive the correct IP on the respective SSID and able to route out to internet fine.


But


All UniFi hardware is no longer able to talk to the controller (192.168.1.1) and the UniFi controller shows them all as offline. They're clearly working though.


And so here I am with the situation of either a), having a working network with no way to manage the UniFi hardware.
Or
b), a broken network where the Waps don't see the vlans and thus don't assign IPs, but I can see all the UniFi hardware in the Controller.


I hope the above makes sense.

Because the Flex Mini switches are somewhat less configurable, I was wanting to use the Primary VLAN ID 1 to manage all the UniFi hardware. The reason for this is that using VLAN ID 1 enables me to choose ports on the Flex Mini switches to be individually tagged or not.
If I choose not to use the primary VLAN ID 1 and use a different VLAN to manage all the UniFi hardware, I loose this ability and can only allow all Vlans or only Allow none per port which will deny me tagging ports as IPCAM vlan and IoT vlan only for endpoint devices.


The reason for using bridges with OPNSense in the first place was to save some money on having to purchase another 10Gb switch. After tinkering with this setup, I'm not honestly sure if this will even make a difference with respect to the issue I'm seeing, because note that ix0 is running out to the shed (to a UniFi switch), and also igb0 is running to the rest of the house (i.e. another UniFi switch).
The OPNSense Router is in the living room behaving as another 10Gb switch.
The router is  a PC with:
Intel i5 7500 @3.40GHz
16GB RAM
ZFS Mirror SSD Boot
1 x 1GB onboard (em0) onboard
4 x 1GB Intel PCIe (igb)
2 x 10GB Intel PCIe (ix)
2 x 10GB SFP+ Chelsio PCIe  (cxgb) ← yet to install


As for firewalls, every bridge is "wide open", in that I've created a rule that says pass on each bridge interface while setting this up:

Code: [Select]

Action: Pass
Interface: Bridge_xyz
Direction: In
TCP/IP: IPv4
Protocol: Any
Source: Any
Destination/Invert: unchecked
Destination: Any

No firewall rules exist nor ip addresses assigned to any vlan or physical interfaces. (i.e. all done on the bridges)
The LAN interface is not presently assigned to anything (it was on interface re0 and assigned)
No IPv6 configured.


Well, I think that is everything, everything that I can think of in terms of information for anyone to explain to me what I'm doing wrong with the MGMT bridge.
All the other bridges work wonderfully well with vlans and physical networks ports and so I'm just not sure why I'm seeing the issue I'm seeing.
I'm fairly sure it's got something to do with how UniFi uses that VLAN ID 1 as I understand it to be somewhat a unique approach compared to other smart switches.


Anyway, if anyone is able to assist, I'd be very grateful to learn more.


(updated: changed attachment to vector so is clearer to read)

[bimbar]

Please buy a small managed switch, should make everything much simpler.

[hoondi]

Hi bimbar,


Thank you, the flex mini switch is a managed device and does allow vlan assignment to ports.


I'm not convinced my issue is purely with the flex mini switches at this stage, especially considering the setup does work with changes to OPNSense only.

[bimbar]

My point was, eliminating the bridges on the opnsense should make everything much less error-prone.

Also, your problem might be with using vlan 1 inside a vlan trunk, that also, in my experience, does not work well. Try using a different vlan tag for that.

[dseven]

I'm not sure if I'm understanding the problem description fully, but...

you cannot (on OPNsense) bridge multiple physical NICs and do VLANs on the bridge - you'd have to add the VLANs to each physical interface individually, then build bridges across the VLAN interfaces.

.... but I agree with @bimbar - get rid of the bridges, and your life will be simpler.....

[hoondi]

oh...


So the bridges I have atm are:

Code: [Select]

bridge0 phy_igb1_Denon, phy_igb2_LGTV, phy_igb3_ATV, vlan_IoT_igb0, vlan_IoT_ix0, vlan_IoT_ix1 
bridge2 phy_ix1_LRM, vlan_Raywood_igb0, vlan_Raywood_ix0
bridge3 vlan_IPCAM_igb0, vlan_IPCAM_ix0
bridge4 vlan_Neighbour_ix0
bridge5 vlan_Guest_ix1
bridge6 vlan_MGMT_ix0, vlan_MGTM_igb0, phy_ix0, phy_igb0

Yes, I have created vlans for each physical interface individually, and yes, I have then created bridges that only include those vlans where the VLAN ID number matches.


However, you're saying that I cannot add multiple physical NICs to a bridge?


Looking at bridge0 for example then, you're saying that I can only have one physical NIC in that bridge? and not 3 like I do atm? (eg: phy_igb1_Denon, phy_igb2_LGTV and phy_igb3_ATV)


oh dear.


Because I have two physical ethernet cables heading in opposite directions, (shed/homelab & the rest of the house) I'm not able bridge these physical ports in conjunction with a vlan?


mmm.

[Patrick M. Hausen]

Yes, I have created vlans for each physical interface individually, and yes, I have then created bridges that only include those vlans where the VLAN ID number matches.

This is how it is supposed to work in FreeBSD. Pretty? No. Working and stable? Yes.

However, you're saying that I cannot add multiple physical NICs to a bridge?

You can. Assuming your ix0 is a trunk port to a switch or an access point, if you bridge e.g. vlan_Neighbour_ix0 with e.g. ix2, then ix2 becomes what a switch would call an access port with PVID "vlan_Neighbour" and no tagged VLANs.

You still would need to move the interface assignment of vlan_Neighbour and consequently the IP address and all services and rules to the bridge interface removing it from vlan_Neighbour_ix0. That vlan_Neighbour_ix0 becomes a tagged layer 2 connection on a trunk and ix2 becomes an untagged layer 2 access port.

Not that difficult, IMHO, and perfectly stable. Also up to and including gigabit speeds the FreeBSD bridge is perfectly capable of holding up to your bandwidth. I'll have to ask Kristof about 10 G 

[hoondi]

okay, after 1am here, so brain is being super challenged, 


Q1.
Bridges 1 through to 5 are okay, because none of the VLANs' parents are any of the included physical NICs, and so that's why they all work fine, correct?


Q2.
Bridge 6 though is in error. i.e. both the VLANs are included with their physical parent NICs in this bridge and this is why I seeing the issue, correct?


If so, I'm kind've scratching my head on what I need to do to fix this still.


what I do know is that physical ports ix0 and igb0 are trunked ports and both physically connect to a trunked port of the switches, being the flex 10GB in the shed and the flex mini at the other end of the house.


So that means I have to take the MGMT vlans out of that bridge? if so, how do I configure these VLANs? do I create another trunk?


I'm thinking I'm way off in the weeds now and so am pausing.


Thank you to everyone for the help on this too, I'm finding this topic extremely interesting.

[dseven]

If you're determined to use bridges, your MGMT network can't be untagged - you'll have to assign a VLAN ID (other than 1) to it, and tag it on the links to your UI switches, and make a bridge containing those VLAN interfaces.

[bimbar]

okay, after 1am here, so brain is being super challenged, 


Q1.
Bridges 1 through to 5 are okay, because none of the VLANs' parents are any of the included physical NICs, and so that's why they all work fine, correct?


Q2.
Bridge 6 though is in error. i.e. both the VLANs are included with their physical parent NICs in this bridge and this is why I seeing the issue, correct?


If so, I'm kind've scratching my head on what I need to do to fix this still.


what I do know is that physical ports ix0 and igb0 are trunked ports and both physically connect to a trunked port of the switches, being the flex 10GB in the shed and the flex mini at the other end of the house.


So that means I have to take the MGMT vlans out of that bridge? if so, how do I configure these VLANs? do I create another trunk?


I'm thinking I'm way off in the weeds now and so am pausing.


Thank you to everyone for the help on this too, I'm finding this topic extremely interesting.

You can not do VLAN 1 over a trunk, as far as I know, so you would either have to use a different vlan id, or do vlan 1 untagged over a different physical interface.

What we usually do is use one port on opnsense to a vlan trunk port on a switch, and not use vlan 1 at all.

[Patrick M. Hausen]

You can not do VLAN 1 over a trunk, as far as I know, so you would either have to use a different vlan id, or do vlan 1 untagged over a different physical interface.

What we usually do is use one port on opnsense to a vlan trunk port on a switch, and not use vlan 1 at all.

Fundamentally VLAN 1 is no different than all others and can be run on a trunk port tagged. I do it at home where I have Mikrotik equipment. But ... Unifi is weird.

So yes, with Unifi we do the same - dedicated untagged port for VLAN 1.

[bimbar]

You can not do VLAN 1 over a trunk, as far as I know, so you would either have to use a different vlan id, or do vlan 1 untagged over a different physical interface.

What we usually do is use one port on opnsense to a vlan trunk port on a switch, and not use vlan 1 at all.

Fundamentally VLAN 1 is no different than all others and can be run on a trunk port tagged. I do it at home where I have Mikrotik equipment. But ... Unifi is weird.

So yes, with Unifi we do the same - dedicated untagged port for VLAN 1.

Usually VLAN 1 is the default native VLAN for a trunk port. The whole thing is just unnecessary trouble.

[hoondi]

mmm,
So my options are:
1. Run a physical parallel ethernet cable to both the shed and the rest of the house purely for management.
or
2. Change the VLAN management to something other than "VLAN ID 1" on the router and all the Unifi switches.


Both are not fun.
The latter more so as these cheap managed flex mini switches limit their capabilities when not doing it the "UniFi" way. (I can't tag individual ports anymore, it's either all or none if I move management off vlanID 1 on these things) and from what I've read, it's agony if you need to re-adopt anything when not using VlanID 1.


mmm, option $3$ it is then. 


Thanks again to everyone helping me with this. I really have learnt heaps and here's what I'm taking away from my homelab endeavours:
1. Keep away from attempting to use OPNSense as a switch with vlans.


2. OPNSense with bridges using physical ports only is simple and works fine, even with 10Gb interfaces. Kind've defeats the purpose of actually using OPNSense (for me), if you're not going to use vlans though.


3. Vendor switch research when used with OPNSense needs more attention than I thought. I like the idea of the UniFi controller but am not keen on the VLANID 1 emphasis so much now.


Thanks again all.

[Patrick M. Hausen]

Just connect OPNsense to one of the switches with one trunk port and one dedicated port for VLAN 1. Then go from switch to switch "the Unifi way".