VLAN/Multiple OPNsense LAN Ports Question

[nVIceman]

I wanted some clarity on this as research hasn't made it 100% clear based on what I am looking to do. I've done VLANs before, but my situation and usage is a bit different.

My firewall has 4 LAN ports. 1 is the WAN, 1 is the LAN, 2 extras unused. 1 Managed Switch. I wanted to implement a VLAN or 2, just wanted to see if it was worth using either of the 2 extra firewall LAN ports to allow better performance or security. It's not a huge deal, but would prefer going the best route from the get go.

I know I can just use the 1 LAN port and have inter VLAN traffic be hairpinned through the firewall as needed. Really, not much inter VLAN traffic, especially not anything taking advantage of high transfer speeds, but would prefer the clarity for future use cases.

Whether the LAN ports are bridged, or simply assigned them for VLAN usage, is there any reason or benefit to use the extra LAN ports when I have just 1 switch, whether it's security or performance?

[Greelan]

As as result of this comment, I now have my VLANs on a separate NIC to the NIC that LAN is on

[guest30640]

Welcome\Hi,

WOW...just read those comments!

I've just recently gone to opnsense from an Edgerouter (both router on a stick configs for home use) but must say documentation is not up to speed in some specific areas and if the above is gospel then that should have been documented about VLANs not being configured on the same NIC as the LAN (native vlan1?)

In my setup I have

igb0 - WAN PPPOE
igb1 - LAN ip:192.168.*.*
igb2 - ip: 10.*.*.* subnets with multiple VLANs (PVID=1, VLANs = T)
igb3 - not used (did try LAGG to switch but threw a wobbly with sensei\laggs\vlans together - which aint good)

So based upon the declaration above about the LAN, do I need to do anything on that igb1 NIC to block anything, currently it has the default FW rule? Since it is not part of the internal VLAN network but I 'assume' that since no VLAN1 has been created (on any interface) that it is still using VLAN1 as the native? Hence why I didn't explicitly setup a VLAN1. EDIT: Just realised I have probably just moved the problem onto igb2 since I use VLAN1 on that interface

[nVIceman]

As as result of this comment, I now have my VLANs on a separate NIC to the NIC that LAN is on

Thanks, but I don't quite understand what that is saying. Don't use VLAN's on main LAN port?

[Superduke]

+1 to this....I was wondering the same.....I have a number of VLANs tagged to my LAN interface and it's been working fine....with firewall rules restricting cross-talk between them and LAN itself....or so I thought....

Is what is being said that general rules allow full access between VLANs on the LAN interface?

And if so, do you need to physically connect an alternate NIC port to create VLANs on them?  I have two spares just sitting dormant but never thought I needed them.....I use Ubiquiti switches and APs.....

[Patrick M. Hausen]

You should never mix tagged and untagged frames on the same interface. Never. Seriously.

[Superduke]

I'm nowhere near competent to understand why.....but if this is indeed true, and I have no reason at all to doubt you.....are all of the Youtube videos that exist on how to setup VLANs on OPN and pFSense wrong in their approach?

If yes....then can you point to a proper approach to separate church from state?

You should never mix tagged and untagged frames on the same interface. Never. Seriously.

[Patrick M. Hausen]

Use one interface for untagged and a separate physical interface for tagged traffic.
A port connected to a switch carrying tagged traffic should carry only tagged traffic.

Of course the idea is not fundamentally wrong or technically impossible. The general advice I give here is mostly about edge cases and possible failure situations. For example you cannot put a VLAN on top of a bridge in FreeBSD. So if you want to have more than one interface in your untagged "LAN" to use the OPNsense device as a cheap switch - bad luck for the tagged VLANs. You can of course put a bridge on top of a VLAN which is the ways this is supposed to work.

Then the IDS/IPS components (Sensei/Suricata) frequently fail in non-intuitive ways in a setup like this.

There are reported cases in which dhcpd does not work on the tagged VLANs if it is also serving the untagged one.

So, it's complicated. I cannot claim to state "it is bad and will not work because ..." - it only has a high probability of "weird" failure modes depending on your setup and hardware.

So repeating my advice: simply don't. Access ports are access ports (untagged only), trunk ports are trunk ports (tagged only). Even Cisco's documentation explicitly states: you should not use VLAN 1 for anything. VLAN 1 is the default untagged VLAN on trunk (tagged) ports.

HTH,
Patrick

[Greelan]

Use one interface for untagged and a separate physical interface for untagged traffic.

I suspect you meant to say “a separate physical interface for tagged traffic”.

I suspect you are probably right that this is largely a concern for edge cases. I’d hope for normal firewall behaviour that carefully defined rules (eg making sure source IPs in the rules are confined to what they should be, rather than using “any”) will avoid unanticipated outcomes. Franco in fact suggests this in his original comment

[Patrick M. Hausen]

Yes, typo - thanks. I fixed it.

[guest30640]

Use one interface for untagged and a separate physical interface for tagged traffic.
A port connected to a switch carrying tagged traffic should carry only tagged traffic.

When I was using an Edgerouter I had all my VLANs including native\vlan1 tagged on the trunk port from the switch to the router but had a dummy VLAN as the PVID, say a vlan 200 (so unwanted untagged traffic would just go to a 'hole'). However, in opnsense I cant get my head round on how do configure opnsense to do that since there appears no way to tag vlan1 and I guess it shouldn't? However, in principle the edgerouter\switch way I was doing it before appears more secure, since VLAN1 was tagged on the trunk with a dummy vlan acting as a PVID?

Side Note to above: VLAN 1 has never been explicitly defined (setup) on my opnsense since I assumed that this would be the default native untagged vlan. All my numerous vlans access ports are defined on the switch, with one trunk (all tagged except vlan1) DHCP from opnsense.

How can replicate this between opnsense and a switch please, if that is the right way? (when i tag vlan 1 @ the switch I then loose connection)

Otherwise guys (who may have to reconfig) anyone wanna start a learning thread on doing this properly because as mentioned there's a lot misinformation on router on stick configs out there?

[Patrick M. Hausen]

If you have an interface, e.g. "igb2", then the untagged VLAN - no matter the number within your larger infrastructure - on that port is simply that: the igb2 interface.

In FreeBSD for every tagged VLAN you create an additional VLAN interface, name it e.g. "vlan27" and set tag 27 and parent interface igb2. That's all there's to it. So you need to create a "vlan1" interface and set the tag to "1" to run it tagged.

If the other end is e.g. a Cisco switch with

Code: [Select]

switchport mode trunk
switchport trunk native VLAN 200then everything that is VLAN 200 somewhere in the rest of your network will arrive untagged and hence on the "igb2" interface on your OPNsense.

So the "don't mix tagged and untagged" advice in the case of OPNsense boils down to: "don't use the 'naked' parent interface for anything on a trunk port, use only the VLANs".

In all my data centers I create a "native-dummy" VLAN that does not carry any traffic, does not contain a single access port, and assign that as native VLAN on all trunks. This way anything a customer might throw into a trunk port untagged does end up in the bit bucket and not somewhere in my management plane or other customer's VLANs.

Did that make it more clear? HTH,
Patrick

[SoCold]

I have a similar setup with a 4 port nic and multiple VLANs. Currently I'm not using the spare 2 ports for anything but my thought was to do a LAG using 1 of the ports and still have a spare. I'm not currently bottlenecked with 1 port but it would be more a learning exercise.

Maybe this is more what you're after? If so you'll need to set it up on both ends, the firewall and the switch.

[Superduke]

Thank you....this helps immensely.  Another question that then spring to mind is APs.  In my case, there is only one (tagged) VLAN I have setup that would require a physical connection directly to the managed switch (for my Xbox's).

My other (tagged) VLANs are all strictly WIFI that deals with IoT gear and the kids with their gear.  So for a given AP, there would certainly multiple VLANs (and LAN) traffic on that one AP physically connected to the one switch port.....and because of my stupidity, I can't see a way around this....lol  Any more thoughts?  I am certainly missing something....

Use one interface for untagged and a separate physical interface for tagged traffic.
A port connected to a switch carrying tagged traffic should carry only tagged traffic.

Of course the idea is not fundamentally wrong or technically impossible. The general advice I give here is mostly about edge cases and possible failure situations. For example you cannot put a VLAN on top of a bridge in FreeBSD. So if you want to have more than one interface in your untagged "LAN" to use the OPNsense device as a cheap switch - bad luck for the tagged VLANs. You can of course put a bridge on top of a VLAN which is the ways this is supposed to work.

Then the IDS/IPS components (Sensei/Suricata) frequently fail in non-intuitive ways in a setup like this.

There are reported cases in which dhcpd does not work on the tagged VLANs if it is also serving the untagged one.

So, it's complicated. I cannot claim to state "it is bad and will not work because ..." - it only has a high probability of "weird" failure modes depending on your setup and hardware.

So repeating my advice: simply don't. Access ports are access ports (untagged only), trunk ports are trunk ports (tagged only). Even Cisco's documentation explicitly states: you should not use VLAN 1 for anything. VLAN 1 is the default untagged VLAN on trunk (tagged) ports.

HTH,
Patrick

[Patrick M. Hausen]

The tagged/untagged distinction really only bothers the two devices that share a trunk port. So if your AP tags VLANs x,y,z while sending VLAN u untagged, and your switch expects VLANs x,y,z tagged and VLAN u untagged, this is absolutely no problem. I would just not use the untagged VLAN at all and create as many tagged ones as necessary, but in the case of a switch and an AP you are less likely to run into trouble than on a firewall.

Inside the switch all VLANs are separate broadcast domains and the frames are switched with tags across the backplane.

Once a frame goes out of another switchport one of separate things happens:

• switchport is an access port assigned to VLAN x and frame is for VLAN x - frame is sent untagged
• switchport is an access port assigned to VLAN x and frame is for VLAN y - frame is never sent out that port, only to ports assigned to VLAN y (and trunk ports)
• switchport is a trunk port, VLAN x is in the list of allowed VLANs (fancy Cisco gear can do that), and VLAN x is not the "native VLAN" - frame is sent tagged
• switchport is a trunk port, VLAN x is the "native VLAN" on that port - frame is sent untagged. But that's what I simply refuse to use. I set the native VLAN to an unused "dummy" VLAN on all my trunk ports so trunk ports only carry tagged traffic.

HTH,
Patrick

P.S. something that just came to my mind and might be helpful:
There is no tagged or untagged VLAN. There are only VLANs - which are separate virtual LANs, broadcast domains, virtual switches if you will. Only once frames of more than one VLAN are transported over one and the same port there are frames that are tagged for VLAN x on that particular port.