VLANs - Multiple LAN ports

[Xigma22]

Hey guys,

I've got a fun one for ya! I have been unable to figure out what I'm doing wrong and could really use some insight here...

I built a little rackmount router with a total of physical interfaces with the following configuration for the interfaces:

re0 - DHCP WAN
re1 - 192.168.15.1/24 - VoIP router dedicated port
bxe0 - Basement Switch (TL-SG3216 v1.0)
bxe1 - 2nd Floor Switch (TL-SG3216 v2.0)

The following have also been created:

bridge0 - comprising bxe0 and bxe1
VLAN1 - 192.168.1.1/24 (parent bxe0)
VLAN10 - 192.168.10.1/24 (parent bxe0)

My network is configured as follows:

|-----Router-----|

bxe0|                       |bxe1

TL-SG3216-B<---->TL-SG3216-2

Disregard the link between the two switches - for the moment, I have disconnected this connection as I haven't had a chance to set up STP.

So here's my problem...

I've been trying to get the VLANs to apply to both switches (bxe0 & bxe1) but I can't seem to pull it off. I've tried adding the VLANs into my bridge interface to see if that would work but everything becomes inaccessible and I have to gui into the router through my VoIP port to change everything back.

Everything works fine out of bxe0 - all VLANs correctly deploy DHCP and all clients can successfully connect.

Aside from that, the bxe1 side of the network is basically inaccessible. I can manually ping from the router to the managed IP address of the bxe1 connected TL-SG3216 switch - but I have no other connectivity.

How does one go about putting the same VLAN in place across multiple LAN interfaces in OPNsense? Does the VLAN need to be on a bridge or..?

I'm so confused by how the VLAN interface assignments work in opnsense - it just doesn't seem very intuitive to me from anything else I've ever worked with.

[Xigma22]

I built a little rackmount router with a total of physical interfaces with the following configuration for the interfaces:

Doh! Should read "...total of 4 physical interfaces..."

[john9527]

How does one go about putting the same VLAN in place across multiple LAN interfaces in OPNsense? Does the VLAN need to be on a bridge or..?

I'm so confused by how the VLAN interface assignments work in opnsense - it just doesn't seem very intuitive to me from anything else I've ever worked with.

I'm new to this as well, but this is what I would do....
- remove the DHCP servers from your current VLAN1 and VLAN10 interfaces
- define two new interfaces for VLAN1 and VLAN10 with parent bxe1
- bridge the two VLAN1 interfaces and assign its DHCP server addresses
- bridge the two VLAN10 interfaces and assign its DHCP server addresses

[Xigma22]

Well...

It made logical sense and I really thought it would work but - no dice.

I'm testing on VLAN 10 without success. No connectivity now but I do get IP addresses assigned from the VLAN 10 DHCP server.

Any ideas..?

[john9527]

Thinking out loud....

Did you adjust your firewall rules accordingly?
By default VLAN1 won't be able to see VLAN10 (is that what you want)?
Add rules to allow VLAN10 to access the internet?

[Oxygen61]

I'm so confused by how the VLAN interface assignments work in opnsense - it just doesn't seem very intuitive to me from anything else I've ever worked with.

It's the other way around... atleast for me it was way to intuitive and i misconfigured everything back then when i first started to configure IEEE802.1Q VLAN Tagging on OPNsense.
But.. fair point.. there is no real documentation, which explains this in detail... meh. :-/

If you want your frames to add an IEEE802.1Q VLAN Tag between Trunk connections (Switch-->Switch Uplinks) or (Firewall --> Switch Uplinks) you will need to configure VLAN Trunk Ports on both sides with tagged VLANs on these both Uplink interfaces.

In OPNsense you do the following:
Interfaces > Other Types > VLAN
Add a VLAN and choose the physical "transport" interface on which you want to carry your Tagged VLANs. You can easily configure LACP/LAGG Interfaces BEFORE adding VLANS to them in order to loadbalance your traffic.
As soon as you added your VLANs (with VLAN Tagg) onto one physical carrier interface you will then see (or add) this VLAN Interface in Interfaces > Assignments.
At this point you should be able to add IP static IP adresses and work just like you would normaly do when configuring a physical interface.
No need to configure any bridge or anything similiar, aslong as a bridge was not intended to exist in the first place.

Forgot to tell about DHCP. When your OPNsense is a so called "Router-on-a-stick" and is doing all the Inter-VLAN-Routing moves AND in addition your DHCP Server is NOT your OPNsense Firewall you will need to configure DHCP-Relay here Services > DHCPv4 > Relay. If you forget to tell your clients how to get to your DHCP Server, without a Relay they would not be able to find the way into a different VLAN Subnet without the help of the DHCP-Relay.

Have a nice day.
Best regards,
Oxy

[Xigma22]

Sorry for the delay in replying, I've been busy with other projects.

Here's a recap of where I'm at.

I want to have VLANs span across multiple LAN interfaces and be able to communicate across those interfaces.

I currently have the following configured:

bxe0 and bxe1 (my LAN interfaces) - Bridged as Bridge0

bxe0, VLAN10_a
bxe1, VLAN10_b
VLAN10_a & VLAN10_b bridged as "VLAN10" Bridge1

The DHCP IP assignments are handed out by VLAN10 under Bridge1

Now... I have a device connected to one of my switches that successfully pulls a DHCP IP address from the VLAN10 DHCP Pool.

But! - That device will not connect to the internet and I cannot ping to it from the router or ping the router from the device. I've set firewall rules on the Bridge as well as the two member VLANs with no success.

I really don't understand why this isn't working.

[Xigma22]

Also,

The only firewall rules I have are the "Default allow LAN to any rule" that allows everything else to communicate correctly.

Does a VLAN need an additional rule or..?

Why is there no documentation for any of this?! 

[Xigma22]

Yeah...

Might be time to scrap this project and buy a finished product.

[franco]

Hi,

"buy".. err... ok... sure.


Cheers,
Franco

[Xigma22]

Yeah.

Buy.

Spend a couple hundred bucks on an out of the box solution that I can have up and running in 15 minutes vs spending untold hours trying to make an undocumented product work. As it stands, I've invested over 16 hours in attempting to get this extremely simple task running with absolutely zero success.

I am out of ideas on how to get this to work and the ideas that have been floated have been unsuccessful. That is not to say that I don't appreciate them. I definitely do. Sadly, the facts of the matter are that I have a network that is screwed, no idea how to fix it, no leads to follow to fix it, and no documentation to engross myself in to even attempt to troubleshoot this issue.

Every time I work with a community supported *nix product, I end up with a bitter taste in my mouth and the overarching feeling of "this is exactly why Linux will never be a mainstream product". I also swear that "this will be the last time"... right up until I try it again.

I've seen nothing to change that view - though I was hoping beyond hope to have an open source solution that actually worked for me.

Thanks for your amazingly insightful post though!

[franco]

Of course, you are welcome.

[elektroinside]

I also spent a couple of thousand euros on my network.
I tried most paid and free solutions after pfsense failed on me. Ended up with OPNsense since it worked best for me.
From the beginning of January, it has become my daily task (I have a goal with this though, I'm not doing it because I'm bored) to learn as much as I can about OPNsense. Where and how it fails (if it does), how it's doing what it does, and how to recover in case of disaster(s) or problem(s).

And even though there are some issues here and there (there's no perfect software in the world, especially OSs), OPNsense generally works very well. I am also considering the facts: relatively young project, opensource, a handful of professionals actively working on this as much as they can, a handful of other users giving as much feedback as they can, and a major upgrade (17.7 -> 18.1). The secret is to get to know the product. This means time spent with it and occasional frustration. But it's the same with paid products as well.

Good luck in your quest Xigma22

[monstermania]

Every time I work with a community supported *nix product, I end up with a bitter taste in my mouth and the overarching feeling of "this is exactly why Linux will never be a mainstream product". I also swear that "this will be the last time"... right up until I try it again.

I've seen nothing to change that view - though I was hoping beyond hope to have an open source solution that actually worked for me.

Hi,
first to say that OPNsense not based on Linux.
It's based on FreeBSD. 
And of course you can buy commercial support for OPNsense!
https://opnsense.org/support-overview/commercial-support/
https://opnsense.org/support-overview/professional-services/
That is much more as many commercial products offer!

This Forum offers free support by other users. Not more. So you can't expect a solution to every problem immediately.

best regards
Dirk

PS: Into our company we use a lot of commercial software. Although we have support contracts, it often takes months to fix known issues!

[Oxygen61]

Hi Xigma22,

i know exactly what you mean, i died many "deaths" back then as i tried to let NAGIOS monitor my OPNsense machine.... it was a complete mess to be honest but it worked in the end. Unix will never be mainstream indeed and there is no need for it to be. It's a niche product for niche applications just like OPNsense to be used by people who spend hours to get this thing to work. On the other hand you have a complete monstrosity called "android" which is a complete mess in itself while being used by millions while also being "*nix OS".

Talking about "commercial/Enterprise-level" support you can always use the commercial support from OPNsense (see the links from monstermania)

As far as i know there is no "out of the box - configure vlan all by itself"-device.
If this would be the case, my current job would be pointless.
The hardest part about VLANs is the theoretical part behind it.
Once you fully understand it's potential, it gets easier.

A few hints and tricks for the future whenever you want to configure a new Unix OS. It always makes a lot of sense to have a valid, already working product at hand, so that in this case you can "play" aslong as you want until the new device, for example OPNsense Firewall, works as expected and can supersede the old device.

A last few things in case you still want to solve the riddle (i will help you aslong as you want):

bxe0 and bxe1 (my LAN interfaces) - Bridged as Bridge0

makes no real sense to bridge two physical interfaces if you THEN afterwards want to logical separate LAN traffic anyway using different VLANs. How far away from each other are these two switches? The point is, i don't think Bridges can even be used as a parent for VLANs. The better way would be to use LACP/LAGG as a parent interface for both VLANs. In this case you would then need to supersede both switches by using a bigger one directly connected to the firewall or use another little "Core" Switch to connect the other both switches trough VLAN Trunk Ports.

Steps:
1. configure LAGG/LACP to create a Interface bundle out of both interfaces bxe0 and bxe1
2. Create new VLAN Interfaces on top of your newly created LAGG Interface Bundle (Parent interface).
3. Configure every VLAN Interface exactly as you would configure a normal physical interface
(IP/Subnet, DHCP, Firewall rules)

First try using just one switch:
|-----Router---------|
      |              |
      | 2 cables |
bxe0+bxe1 LACP on both sides
      |              |
[---48-Port Switch---]


second try using a "core" to connect the other switches:
|-----Router---------|
      |              |
      | 2 cables |
bxe0+bxe1 LACP on both sides
      |              |
 [---16-Port Switch---]
      |                      |
[-TL-SG3216-B-]  [-TL-SG3216-2-]

My drawing is sh*t but i hope you get the idea. Or just just forget VLAN and just use the bridge all by itself. But then you won't be able to seperate traffic using different VLAN subnets....

@monstermania

first to say that OPNsense not based on Linux.

What he meant was probably "Unix" and not Linux