Solved: help for having VLANs setup across spare router ports

[hoondi]

Hi All,
Have read a couple posts from a couple of years ago (Mainly Patrick's responses) about this but I'm afraid I'm just too dumb to understand and need bigger brains to help me if anyone's willing.

So far, I've got OPNSense running beautifully using a single port for WAN and a single Port for LAN that includes 4 VLANs. What I've since learnt is that TrueNAS is an absolute *&%^*$ if you want to spin up services with it on your desired VLANs.

Anyway, Rather than buy another Ubiquity Flex 10 Gig switch in such lovely economic times, to in order to bash TrueNAS over the head and comply, I'm wondering if I can utilise the existing ports on the router (i5-7500 CPU @ 3.40GHz, 32GB RAM) to have a few house bound devices hang off it so I can move the Flex switch to the shed.

I know, I know, I shouldn't be switching on a router. and yes, if throughput turns to crap, I'll starve the family for a month and get another Flex 10GB switch.

Anyway, I spent fifty-hundred hours putting the following diagram together which is an example of what I'm after, but simplified with just 2 VLANs ← hopefully am not too dumb and can scale out the rest things click for me.

The grouping of opt ports on the router symbolise installed PCIe NICs and the thicker lines for opt5/opt6 ports = 10 gig NIC.

I've already created a bridge with all the ports, but the VLANs are still configured to only use opt1 as their parent, which is how I had it originally configued before attempting this. I can tear down the router and start again no probs if need be, but it would ideally be nice to preserve it since I've got Unify and Arguard up and running on the router via plugins.
 
Obviously can provide more info if needed, but figured the diagram below is what a lot of home-lab users would like to try/test when physical/locality issues come into play.

If anyone is able to tackle a howto to achieve the diagram below, I'm sure I wouldn't be the only appreciative one.

Thanks,
 
(edit: I tried to include the diagram inline using asci art but it was too wide and crapped out... soz)

[Patrick M. Hausen]

I've already created a bridge with all the ports, but the VLANs are still configured to only use opt1 as their parent, which is how I had it originally configued before attempting this. I can tear down the router and start again no probs if need be, but it would ideally be nice to preserve it since I've got Unify and Arguard up and running on the router via plugins.

It works the other way round.

You need to create a VLAN interface with tag 1 on each physical interface where you need them, then create a bridge with all these VLAN interaces as members. Same for VLAN 10 and so forth. One bridge per VLAN. Each VLAN explicitly created on each port - not on the bridge.

[hoondi]

ah...

I think it just clicked for me.

Will give it a go at a more civil hour and I think I understand now.

Thank you.

[hoondi]

update: This post has the bridges wrong, so jump past this. Am leaving it here so others can learn from my mistakes. 

[Patrick M. Hausen]

eg:
bridge 1 members consist of: vlan1_em1
bridge 2 members consist of: vlan1_em2
bridge 3 members consist of: vlan1_em3 vlan10_em3
bridge 4 members consist of: vlan1_em4
bridge 5 members consist of: vlan1_cx0
bridge 6 members consist of: vlan1_cx1 vlan10_cx1

Nope.

bridge1: vlan1_em1, vlan_em2, vlan1_em3, vlan1_em4, vlan1_cx0, vlan1_cx1
bridge10: vlan10_em3, vlan10_cx1

One bridge per VLAN as I wrote already 

You want all VLAN X instances across all physical ports connected, not all different VLANs on a single physical port, right?

[hoondi]

Ah...

So that's why I wasn't understanding the gateway IP and thus DHCP setup.

I was so sure I had it, 

okay,

So I've now got the bridges setup. I actually have 5 VLANs in total. (see attached), assigned the bridges as the gateway IP ending in dot one for each VLAN and the DHCP setup makes sense now too.

Thank you Patrick. You are a champion!

[Patrick M. Hausen]

If you intend to use IPv6, enable "link local" on the bridge.

[hoondi]

Thank you,
I've switched all iPv6 off where possible so this is about the only thing I got right. ha!

Thanks again.

[hoondi]

So not quite happening.

with tunables enabled:
System → Settings → Tunables:
net.link.bridge.pfil_member to 0
net.link.bridge.pfil_bridge to 1

I've got the bridges setup to group together the vlans I've created for each physical port.
(Notation used when creating the vlans is "parent-port_vlan-number")
bridge0   cxl0_141, cxl1_141, em0_141, em1_141, em2_141, em3_141, ix0_141, ix1_141
bridge1   cxl0_145, cxl1_145, ix0_145, ix1_145

Subnets are as follows where opnsense web management is 192.168.140.1 ← all Ubiquiti hardware running on this subnet for management as well.
untagged = 192.168.140/24
vlan141 = 192.168.141.0/24
vlan145 = 192.168.145.0.24

I've assigned IPs to the bridges:
bridge0 = 192.168.141.1
bridge1 = 192.168.145.1


DHCP is active and enabled for each of the above, being the untagged/LAN and the two bridges:
ISC DHCPv4   
  [140_MGMT]
  [141_IoT]
  [145_Raywood]
this is working as expected and all devices are receiving respective IP addresses on the correct vlan when using the single physical ethernet port of ix0 (← LAN) connected to the Ubiquiti switch which is also setup for vlans.


The moment I unplug a device from the Ubiquiti switch where it is receiving 192.168.141.8 (an AppleTV with static assignment) and plug it directly into the em2 ethernet port which is a spare port configured for use with vlan141 in the bridge on the opnsense router, I get nothing. (i.e. a 169.254.x.y).

Q1.
em2 is the parent for vlan141 only and is in one bridge only being bridge0 and so I'm expecting the Apple TV to receive an IP of 192.168.141.8, ← I'm expecting the Apple TV to receive an IP from the only vlan  running on the wire?

Manually assigning an IP doesn't help and so there's more than just DHCP not happening.

Q2.
Enable Interface is checked for all entries in Interfaces → Assignments.
Do all need to be enabled? or just the vlans and the bridges?


I see that floating rules (about 19 of them) for the untagged and two bridges contain two entries for DHCP.
But what I'm seeing in the live view of the firewall log for the physical port em2 is:
em2      2024-07-15T00:12:14   0.0.0.0:68   255.255.255.255:67   udp   Default deny / state violation rule
and so it looks like that not only is the firewall preventing DHCP, there's more going on as  I've attempted a manual ip config and still have no luck, I'm starting to wonder if I've set things up incorrectly still.

I only have two firewall rules on each of the two bridges and the untagged subnet.
Basically pass DNS and !PrivateNetworks:
IPv4 TCP/UDP   141_IoT net   *   141_IoT address   53 (DNS)   *   *      Allow DNS      
IPv4 *   141_IoT net   *   ! PrivateNetworks    *   *   *      All Access to Only Internet


Q3.
As the Firewall shows an entry for the raw port, the vlan(s) created on the raw port and the bridge grouping all the vlans together, do I need to apply the above rules to all 3 mentioned or just the bridge? ← I've only configured firewall rules for the bridges and untagged/LAN.

To test, I've created firewall rules on em2 and em2_vlan141 to pass all in and out which has not helped. 0.0.0.0:68 is no longer being blocked on em2 though.
I also disabled !PrivateNetworks for the 141 bridge which has not helped either and so I'm all out of ideas.

I'm clearly missing something and so am wondering if anyone with a bigger brain can spot my error.

ta.

[Patrick M. Hausen]

I've got the bridges setup to group together the vlans I've created for each physical port.
(Notation used when creating the vlans is "parent-port_vlan-number")
bridge0   cxl0_141, cxl1_141, em0_141, em1_141, em2_141, em3_141, ix0_141, ix1_141
bridge1   cxl0_145, cxl1_145, ix0_145, ix1_145

Looks good.

The moment I unplug a device from the Ubiquiti switch where it is receiving 192.168.141.8 (an AppleTV with static assignment) and plug it directly into the em2 ethernet port which is a spare port configured for use with vlan141 in the bridge on the opnsense router, I get nothing. (i.e. a 169.254.x.y).

Of course. VLAN 141 on em2 is tagged. Tagged VLANs are for links between routers, switches, servers, ...

An end device like an Apple TV needs an untagged port. An untagged port carries one VLAN only. To turn em2 into an untagged port in VLAN 141:

- remove the em2_vlan141 interface
- add em2 to the bridge for VLAN 141

em2 is the parent for vlan141 only and is in one bridge only being bridge0 and so I'm expecting the Apple TV to receive an IP of 192.168.141.8, ← I'm expecting the Apple TV to receive an IP from the only vlan  running on the wire?

This is not how this works. In your current configuration the other end needs to understand and use VLAN tags, too, so it needs to be a switch or similar. Reconfigure em2 like I outlined above.

Let's get this fundamental issue fixed, first, before we dig deeper into firewall rules.

Explicitly assigning and enabling the parent interfaces should not be necessary, anymore, if you create VLANs on top of them. It will be necessary for e.g. em2 if you make it a member of the bridge in untagged fashion.

[hoondi]

well,

I did say I need a bigger brain right at the start! hehe

You've blown my mind with this fundamental nugget.

I always thought that a switch would assign/tag the actual/physical ports with a vlan (or vlans) for endpoint devices, andd that if only one vlan was configured, that's the IP/subnet that the endpoint would utilise.

I now know that sounds stupid when I read it back!

I'll have another go when I'm back there tonight.

Thanks again Patrick, I'm very appreciative of your knowledge and your willingness to share.

[hoondi]

Hi all,

I managed to get the bridges up and running with 5 vlans and so thank you Patrick.

I started over with 24.7 in the end.

I've since noticed that Unifi and AdGuard install, but I'm struggling to identify what interface they're running.
I posted over at https://forum.opnsense.org/index.php?topic=41803.0 thinking it might be a 24.7 issue, but I'm now suspecting the issue is to do with the bridges.

I have a spare box and so will test a basic box using 24.7 with WAN/LAN only.

anyone else using Unifi plugin with a bridge setup by chance?

[Patrick M. Hausen]

How many and which kind of Unifi devices do you have and how are you intending to connect them? Are you planning to use VLANs with Unifi APs and map them to SSIDs? If yes, you need a managed switch, period. While you can get far with "switch emulation" with bridges that doesn't play nicely with Unifi. If you have only one WiFi network and SSID, it will work without a switch.

I'll explain the details after your answer.

Kind regards,
Patrick

[hoondi]

Hi Patrick,

2 x U6Pro
3 x Flex-Mini
1 x Flex-XG      

as of 24.1, I had everything connected via a single 10GB copper port on OPNSense that goes to the Flex-XG and then branches out from there.
I've got two SSIDs running which work fine (vlan141 and vlan145) and have configured the respective ports for the vlans and I can't fault it. My setup provides the correct vlan to the device in question and DHCP is issuing the correct IPs etc etc.

Then I started over with 24.7. Please note that I have not altered any physical port connections, nor have I changed anything in Unify at the time. So all the vlans and devices are happily functioning fine and as expected on their respective vlans and ports.

It's just that after installing Unifi and starting the daemon via the plugin section, I'm not able to get to it on port 8080 to set it up.
As my MGMT/LAN (140) network is able to access all others atm, I expected Unify to appear on 140 subnet also. As that wasn't the case, I've since tried:
192.168.140.1:8080
192.168.141.1:8080
192.168.142.1:8080
192.168.143.1:8080
192.168.144.1:8080
192.168.145.1:8080

no cigar...

I've got more info over at: https://forum.opnsense.org/index.php?topic=41803.msg205501#msg205501 when I thought it was a 24.7 issue.

Update:
I've since installed AGH and I can't get to it on port 3000 either, i.e. I've tried all my vlan gateway IPs as well to no avail.

Just to be clear, all my Unifi equipment is functioning as expected atm, as I haven't even looked at removing the 10GB switch from the house and move it to the shed/home-lab yet. ← I wanted to get it all happening first before physically moving anything.

I can send screenshots if you're after more info.
I've attached what the Unifi controller "tree view" was showing before I started over.

The only difference from I can see from when it was working/accessible to not now is bridges and the new OPNSense version.

I'm writing the new 24.7 to USB stick now to test as well.

ta.

[Patrick M. Hausen]

OK, I probably cannot help with the controller proper. I run it in a Linux VM and not on OPNsense.

The intention of my questions was that Unifi expects VLAN 1 to be untagged for provisioning which conflicts with the general advice not to mix untagged and tagged frames on the same port on OPNsense.

That's why I run a trunk port with only tagged VLANs and another dedicated port for VLAN 1 from my OPNsense to my Unifi switch.

Kind regards,
Patrick