Mixed untagged and vlan tagged networks, Need help correcting my mistake

[wiggler]

Hello all! I'm looking for some guidance on my current home network setup. It consists of an opnsense firewall, a slew of unmanaged switched, and a unifi access point. For the most part everything works great, but I love to tinker with my setup, and that of course leads to self-inflicted issues.

Originally, my main LAN subnet was all untagged traffic, and I added a Guest VLAN for use with the unifi AP, this worked perfectly fine for months, as I originally setup my guest subnet to only use IPv4. A few weeks ago, I started actually digging into IPv6, and I learned that I could request a smaller IPv6 prefix from my ISP and assign different prefixes to each of my networks. I was able to enable IPv6 for my guests. Nice!

Recently, I was using my main windows PC and happened to check my network info and noticed, along with the IPv6 addresses from the main LAN network, it was getting an IPv6 address from my guest network as well. This is not the behavior I expected, so I began looking into the cause.

It's my understanding that the reason my windows PC was getting an IPv6 address for my guest network, is the mixing of an untagged main network and a vlan tagged guest network has allowed the guest router advertisement to leak into the main network. Not ideal!

I'm trying to find the best way to prevent this leak. Do I need to create another VLAN for the main network? If the unifi AP is still tagging the guest network, will that traffic get retagged as the main network? Do I need a managed switch for the AP so I can configure a trunk port?

Thanks in advance!

[OPNenthu]

Enabling IPv6 can be an eye-opener to existing L2 issues.  It's not clear how you have things wired but it does sound like you need a VLAN-aware managed switch.  Probably the issue was there before but you didn't notice under IPv4 because DHCP is centrally managed and you were only getting one IP.

Right now OPNSense and the AP have no idea that your switch isn't enforcing VLANs.  From the perspective of OPNsense you have two networks (a native LAN and a tagged Guest) so it does what an IPv6 router should do and multicasts RAs on each interface.  Likewise the AP is probably tagging Guest frames as it should, but it's your switch's responsibility to make sure they aren't forwarded out the wrong ports.  Unmanaged switches don't do anything with VLAN tags; they just get passed through.

Depending how you've connected things, the untagged and tagged traffic are sharing wires and switches on the same broadcast domain.  Since the IPv6 RAs are multicast, the ones from the Guest network are also visible to the Windows PC and subject to how that particular client handles tagged frames.

Let us know how you've wired things, but certainly a managed switch should help.

[wiggler]

My opnsense firewall is connected to a tplink 2.5gb switch we'll call switch A. Connected to switch A, is a netgear 1gb PoE switch (switch B) to which the unifi AP is connected.

My windows pc is connected to another tplink 2.5gb switch (switch C), which is also connected to switch A.

The firewall and switches A and B are in a network closet with the modem, along with a bundle of ethernet cables that go throughout the house. There are a number of other unmanaged switches connected to switch A in other rooms, but switches A, B, and C cover the relevant devices.


I'm thinking of replacing the threes switches in the network closet (there is a third non-PoE 1gb switch in there as well. I know, it's a mess) with a 16 port managed switch. I was looking at the Ubiquiti USW-Pro-Max-16-PoE, which should cover my needs, and I'm already running the Unifi Network Application in a VM for the access point, but I'm certainly open to suggestions.

[meyergru]

Ironically, in contrast to the thread title and what is normally considered best practice on OpnSense, you will probably end up mixing tagged and untagged traffic again when you turn to Unifi equipment - just for practical reasons.

Take a look at this to understand what I mean: https://forum.opnsense.org/index.php?topic=48522

[wiggler]

Of course nothing can be simple! I was poking around the unifi software and did notice there was no option to change the VLAN for the management (default) network, so I starting to wonder about that.

The only guest network clients will be connected to the unifi access point, and my plan was to connect it to the unifi managed switch. Do you think I would be OK leaving the network configured as is (with an untagged lan network and vlan tagged network) and rely on the managed switch to keep the guest network contained? I assume I would have to set the access point's port for both untagged and guest vlan tagged packets.

[meyergru]

Of course nothing can be simple! I was poking around the unifi software and did notice there was no option to change the VLAN for the management (default) network, so I starting to wonder about that.

There actually is that option and you can use it. However, when you have to factory reset a Unifi device or want to adopt a new one, it will work only on an untagged network port first. So it is a matter of practicality to not even try using a VLAN for management. All of that is explained in the linked thread.


Mixing tagged and untagged traffic on the same interface is no problem for Unifi. It can be a problem for FreeBSD/OpnSense, because sometimes, it cannot discriminate between traffic on VLANs and the (untagged) parent interface. This can lead to subtle problems with monitoring to really mixing traffic between them with certain buggy NIC drivers, for which reason it is generally better avoided.

I am not into the specifics, @Patrick M. Hausen can explain it better.

For my needs, I chose to mix tagged and untagged because of Unifi - on the other hand, I do not use any features that could be problematic and also, my NICs do not seem to expose any idiosyncrasies.

[wiggler]

So the main issue is that I'm mixing my untagged main network and tagged guest on the same interface on my firewall. Which is probably causing my IPv6 router advertisements to leak into the main network.

My firewall has 4 NICs, of which only 2 are in use as WAN and LAN (which includes the tagged guest network). Does that mean instead of parenting my guest network to the same NIC as the main LAN network, I could use one of the unused NICs on the firewall exclusively for the tagged guest network, and then plug it into one unmanaged switches and hope for the best?

I suppose it would be worth a shot, since it wouldn't cost me anything, besides one more ethernet cable. Even if I do get a managed switch this would probably be the preferable configuration since it would avoid mixing untagged and tagged networks on one interface at the firewall, correct?

[Patrick M. Hausen]

I suppose it would be worth a shot, since it wouldn't cost me anything, besides one more ethernet cable. Even if I do get a managed switch this would probably be the preferable configuration since it would avoid mixing untagged and tagged networks on one interface at the firewall, correct?

Correct. And if you have the ports, it's just one cable, so go ahead.

[OPNenthu]

Do you think I would be OK leaving the network configured as is (with an untagged lan network and vlan tagged network) and rely on the managed switch to keep the guest network contained?

In either case, whether you use VLANs or separate native networks, you will still need firewall rules to restrict traffic between the networks.  I'm assuming you know that but you mentioned something about containment here, so just worth mentioning.

[wiggler]

I gave it a shot. Unfortunately it seems like one of the switches is the culprit leaking the guest network into the lan network. As my windows PC is still getting a guest net IPv6 address. Looks like I'll have to invest in a larger managed switch to replace the three unmanaged switches in my network closet.

On the bright side, I'll have something new to tinker with.

@OPNethu, I already had a rule for the guest network to block any traffic heading to the lan network, but not the other way. Could adding a rule blocking lan from reaching the guest network prevent lan devices from getting guest addresses? I'll give it a try.

[OPNenthu]

You mentioned earlier that you had switch C connected to switch A.  Did you try separating those as well?  Put Switch A on one of the router ports and put Switch C (with the Windows PC) on the other router port.  Configure them as separate networks in OPNsense.  Make sure those switches don't link to each other.  Also, try disconnecting the Ethernet cable from the PC and plugging it in again in to reset the connection, in case the IPv6 addresses are sticking around.

Could adding a rule blocking lan from reaching the guest network prevent lan devices from getting guest addresses? I'll give it a try.

I don't think so, because IPv6 RAs are part of the ICMPv6 protocol which is enabled by default in OPNsense via system rules.  You can't disable or override those.  However, even if you managed to you would only be masking the issue.  I think that by having those switches all connected together you had a single broadcast domain and that needs to be sorted out, IMHO.

EDIT: attaching a sample diagram with some made-up IPs.

[wiggler]

You want me to bridge switches A and C through 2 ports on the firewall?

I've power cycled the windows machine and when it first comes up it gets an IPv6 address from the main untagged network right away, but after a minute it will get an address for the guest network. And you are right, the rule did nothing to help.

I think I'm going to need a managed switch to keep the guest network from leaking. Since the traffic for the guest network is only between the unifi access point and firewall, it should be simple.

[OPNenthu]

You want me to bridge switches A and C through 2 ports on the firewall?

No.  No bridging.

Those are to be separate routed interfaces.  Check the diagram I just added to my last post.

[wiggler]

OK I see. I think that would work, but switch A goes out to a bunch of other rooms (with switches of their own) throughout the house, switch C being one of them. I would like them all on the same main subnet, except the guest traffic.

Are you suggesting to split the network to try to isolate the leak? In that case, would it be best to put switch C, with the access point on the separate port? Then that would have the guest subnet, and a sort of secondary lan subnet?

[OPNenthu]

Are you suggesting to split the network to try to isolate the leak?

Yes, but that's required unless you're using VLANs.   What VLANs bring to the table is that they create virtual L2 domains, which allows you to create networks atop shared physical infrastructure.  Without them (if you're using native networks) then you need separate ports / switches / cabling for each network in order to maintain L2 isolation.

So unless you switch to VLANs, you need to keep those networks physically separate.  It's not a hard requirement as you already know, it's possible to use the same L2 domain, but you'll see the kind of issues that you're seeing.

In that case, would it be best to put switch C, with the access point on the separate port?

Yes

Then that would have the guest subnet, and a sort of secondary lan subnet?

Not sure where you're getting this "secondary" lan from?  There would only be two native networks: the main LAN (router port 1), and Guest (router port 2).

You might be thinking that the AP needs both a native network as well as a tag for Guest?  If so, I think that only applies when you are using the UniFi switch with VLANs.  If you're not using VLANs then there won't be any tagged network- just the native Guest net.

To be fair, I've never set up a UniFi AP this way so I'm speaking a little bit out of my rear end.  I *think* you can use it like this.  Someone should correct me if not :)