Questions for a migration from UniFi USG-Pro

[Aardvark Salad]

Hey, folks!

I’ve got a UniFi network and I’d like to replace my old USG-Pro-4 with an OPNSense installation running on a new Protectli VP2420. The house is three levels, and I’m using the existing coax cable wiring with 2.5gig MoCA adapters to provide signal between floors. The cable modem, USG-Pro-4, and a US-16-150W switch are in the basement, with the LAN-side of the firewall running back upstairs on the same coax via MoCA to US-8-60W switch. That switch runs a WAP, and another MoCA connection runs to another WAP on the upper floor to provide a wireless backhaul for a location I just can’t get a cable to.

I’d like to move the cable modem to the main floor, feed the WAN port of the new VP2420, and move each of those MoCA connections to the remaining three LAN ports. The new firewall would act as the inter-VLAN router (as the USG does now), and I’ve configured the VLANs on bridges for all of the LAN ports. The USG doesn’t do any VLAN separation at this point, though I plan on adding it in at a later date. VLANs on the UniFi gear all have gateways (the .1 address), but most the documentation I’ve been reading for OPNSense says to not set up gateways for your VLANs in most cases.

I’m essentially trying to swap the new device for the old device without having to reconfigure anything, but I’m not sure I’ve got everything figured out. The two main questions I have at this point are:

• Do I duplicate the VLAN gateway IPs from the UniFi gear onto OPNSense?
• If I use up all of the LAN interfaces with VLAN bindings, will the GUI still function?

And as a more broad question, I’m thinking it’d be better to start over and simplify the network by collapsing a bunch of the /24 networks into a couple of /23’s for wireless, wired, guest, and restricted IoT and expand later if necessary.

Thanks!

[dseven]

What are the VLANs used for?

What do you mean by gateways on VLANs? Are you talking about the firewall itself having an IP address on each VLAN's subnet? Of course you need that if the firewall is to act as a router. Or are you talking about some upstream gateway? What would that be for?

What do you mean by "VLAN separation"? VLANs do separation by nature. Maybe it'd be more clear if you describe what the VLANs are for.

What do you mean by "VLAN binding", and how would you "use up" LAN interfaces with them?

You mention "collapsing a bunch of the /24 networks" - what are those used for (why do they exist)?

Note that the ports in your new Protectli box are not switch ports - they are individual NICs. You could do software bridging across them, but it's not very efficient.

The usual configuration would be one physical LAN port, connected to a physical switch, with whatever VLANs you need, tagged.

[Aardvark Salad]

What are the VLANs used for?

With the exception of the guest network, right now they’re just directly mapped to /24 subnets. Eventually I plan on isolating traffic, keeping the lab from screwing up the rest of the network, segregating wireless and wired traffic, etc.

What do you mean by gateways on VLANs? Are you talking about the firewall itself having an IP address on each VLAN's subnet? Of course you need that if the firewall is to act as a router. Or are you talking about some upstream gateway? What would that be for?

This is where I think learning VLANs with UniFi caused me some problems. When I configured VLANs on my UniFi switches I had to configure each VLAN with a “Gateway IP/Subnet” ; “Wired” is 10.10.10.1/24 and “Wireless” is 10.10.15.1/24, for example, with the untagged/1 VLAN having the 10.10.1.1/24 gateway and network. Each VLAN has a DHCP forwarder to a ISC DHCP server. UniFi also seems to not restrict inter-VLAN traffic as a default, which is very different from (I think) every other piece of gear.

The switches I have, as far as I know, do not have any routing capability so all traffic heads back to the firewall/router appliance for inter-VLAN routing.

What do you mean by "VLAN separation"? VLANs do separation by nature. Maybe it'd be more clear if you describe what the VLANs are for.

Yeah, bad terminology again on my part. I’m not sure what the OPNSense equivalent would be. I’m guessing ACL, firewall rules, or maybe some policy? As mentioned above, all VLANs on the UniFi gear can talk to any address on the network unless additional firewall rules are added.

VLANs right now are almost entirely just segregation of traffic types and devices, with the eventual goal of ramping up network isolation for things like IoT that can hit the internet (maybe a HomePod) and stuff that shouldn’t (Kasa Switches, for example.) There’s a Docker cluster in there that is assigned a /24 that’s handled by ProxMox. Stuff like that.

What do you mean by "VLAN binding", and how would you "use up" LAN interfaces with them?

What’s the right term here? Each of the three LAN interfaces is a parent to the VLAN tag (vlan10 to igc0, igc2, igc3 for example), and each of those VLANs is attached to a corresponding bridge (bridge0, vlan10 has igc0, igc2, and igc3 participating). igc0 also has a static IP currently for the web GUI, and non of the bridges or VLANs have been actually activated.

You mention "collapsing a bunch of the /24 networks" - what are those used for (why do they exist)?

I’ve got a bunch of subnets configured to attempt to logically group devices; wired goes on .10, all Docker traffic goes on .30, my funky wireless backhaul to the workshop is on .17, etc.

I absolutely could just shove everything on 10.0.0.0/8 and just abandon the VLANs right now with a little bit of effort; I’d need to collapse all of the DHCP configurations and probably nuke all of the VLAN configuration on the switches and access points.

They exist right now in a “planning” state. I’ve been slowly trying to learn the ins and outs VLANs, isolating networks, and the like, and that’s largely prompted my move to OPNSense. The UniFi gear is good on the interface part, but they tend to mangle the terminology and make “standard” things like VLAN isolation more difficult than it should be; they don’t use “Trunk” or “Trunking” anywhere in their documentation, for example, so mapping industry standard terminology back is an extra step that I just don’t want any more.

Note that the ports in your new Protectli box are not switch ports - they are individual NICs. You could do software bridging across them, but it's not very efficient.

The usual configuration would be one physical LAN port, connected to a physical switch, with whatever VLANs you need, tagged.

Yep, I’m aware of that, hence the bridging configuration I’ve set up. I should have enough horsepower with the new hardware to make it quite a bit faster than the 8ish-year-old USG-Pro-4 and be able to run IDS/IPS that I can actually have some control over even with the software switching. The eventual plan is to replace the old switches with newer devices to offload all of the extra overhead, but the budget isn’t here yet.

In the current environment, though, I think I still need to have at least two interfaces bridged for the downstairs and main floor switches; not being able to get a Cat5/6 pull between floors means I’m stuck with the single coax runs between the main, basement, and upper floor..

Although, maybe I could just run a single trunk interface to the 8-port desktop switch and add another trunk from the 8-port to the downstairs switch. In that case, do I configure the GUI to use one of the extra 2 interfaces?

[dseven]

I would recommend avoiding the bridges if you can. IMO it adds complexity that you don't really need, and it's not efficient.

Also, I don't think you can bridge physical interfaces and also VLANs on those interfaces at the same time - e.g. you can't have a bridge with igc0 thru igc2 and another bridge with VLAN 10 on each of those three ports...
so you would have to make all LAN networks (including the one used for management) tagged VLANs.

I would recommend one LAN port, with all your VLANs tagged, connected to a managed switch. The VLAN interfaces on opnsense would each be configured with the ".1" IP address for their respective subnets (I think that addresses your first question). You could actually leave your management network untagged in this case, although some people will tell you that that's invalid, you should never do it, freebsd doesn't support it, etc... but it works for me. The only issue I've had with it is where a Windows PC on a switch port with the untagged and tagged VLANs present sees IPv6 router advertisements from all VLANs, but [some] Windows NIC drivers stupidly (IMO) strip VLAN headers from packets and blindly accept them... but that's not really a freebed/opnsense issue (and can be avoided by configuring a managed switch to only present the untagged VLAN to the Windows NIC.

I still don't really understand your second question. VLANs don't "use up" physical interfaces, unless you mean in the context of my second paragraph above (re. mixing untagged and tagged bridges).

[Aardvark Salad]

Thanks for the follow-ups. The more I’m looking at this, the more I’m thinking I just need to nuke it from orbit and start over with a phased approach while I unlearn the UniFi way of things with more complex network topologies.

What I was originally envisioning was using the OPNSense box as a firewall/router/core switch, and moving the UniFi switches out to the edge. Right now, it’s a weird configuration with the cable ingress going down to the basement, into the modem, then to the USG-Pro-4, followed by the 16-port switch, and then back upstairs through the same ingress cable via a pair of MoCA adapters. The cable company seems to think the MoCA connection on their side of the cable feed indicates old equipment and constantly sends resets to the modem. If I shift to OPNSense as the core switch, my understanding is that I need to bridge at least two interfaces if I want to keep the VLAN topology in place.

Right now, I’m thinking that I just simplify things: pull out the split-DNS and DHCP server and move to the OPNSense internal offerings (UniFi didn’t properly map dynamic hostnames), nuke the ten-ish existing VLANs and collapse the IP space down to an untagged /23 or /22 network, and treat the small 8-port managed switch as a hybrid core/edge switch to feed both the downstairs (my office/lab) 16-port device and the couple of WAPs and miscellaneous devices near the 8-port.

As for this part:

I still don't really understand your second question. VLANs don't "use up" physical interfaces, unless you mean in the context of my second paragraph above (re. mixing untagged and tagged bridges).

“Use up” is again my lack of proper understanding of terminology. The VLANs (in the prior thinking) would be associated to bridges, and the bridges would be associated to physical interfaces. My understanding was that I can’t have the VLAN bridges plus the untagged network on a physical interface and then add the local machine’s IP for the GUI also attached to that physical interface. The more I think about it, though, the more I’m realizing that’s probably incorrect; the OPNSense gateway address should also be the GUI address. Where I’m getting lost, I think, is how all of that would transition from three trunk ports to the router’s gateway interface. It’s immaterial now, since I’m going to redesign this whole bunch of spaghetti.

UniFi was great for making the “Oh look, a new toy!” phase very accessible. I’m now in the “Hey, maybe I shouldn’t have played with that toy so much” phase of the learning. 

Thanks a bunch for the patience in helping me work through my abundant terminology gaps and driving to an actual workable solution!

[Patrick M. Hausen]

“Use up” is again my lack of proper understanding of terminology. The VLANs (in the prior thinking) would be associated to bridges, and the bridges would be associated to physical interfaces.

If you are referring to OPNsense/FreeBSD you got it the wrong way round. You need to create the VLANs on the physical interfaces and then bridge the VLANs - one bridge per VLAN - if you really need to turn your OPNsense into a "switch".

So e.g. on OPNsense:

vlan0002 - ix0, tag 2
vlan0003 - ix0, tag 3
vlan0102 - ix1, tag 2
vlan0103 - ix1, tag 3

bridge02 - vlan0002, vlan0102
bridge03 - vlan0003, vlan0103

...

I would connect the OPNsense with a redundant (LAGG) link to one of the existing switches with all VLANs on top of that LAGG interface, then connect the other switches to that first switch or in a daisy chain.

[Aardvark Salad]

I would connect the OPNsense with a redundant (LAGG) link to one of the existing switches with all VLANs on top of that LAGG interface, then connect the other switches to that first switch or in a daisy chain.

I’ll need to check and see if the smaller switch supports LAGG, but would that work? The current firewall/router is handling all of the inter-VLAN routing since the switches are managed L2 only; UniFi set up all of the inter-VLAN stuff transparently. I think I’d need to set up static routes from all of the subnet’s gateways to the OPNSense LAN gateway.

EDIT: As a question, if I just trunk to the first switch how does traffic get from the 10.10.20.1 (VLAN) gateway to the 10.10.0.1 (OPNSense) gateway, or route between VLANs? My gut is saying this is all of the stuff the UniFi gear was hiding from me.

[Patrick M. Hausen]

OPNsense can carry all the VLANs on a trunk port and take care of all the inter VLAN traffic. In most cases that is the desired topology because "firewall".

It only gets complicated when you want to connect multiple switches, each with a trunk port, to OPNsense. Then you need to mess around with bridge interfaces like I hinted at above.

With a single trunk, LAGG or single interface, into OPNsense you can have as many VLANs as you practically want and have OPNsense route and filter between them. The switches need only be layer 2, of course in that case. Just carry the VLANs via trunks from switch to switch.

[Aardvark Salad]

With a single trunk, LAGG or single interface, into OPNsense you can have as many VLANs as you practically want and have OPNsense route and filter between them. The switches need only be layer 2, of course in that case. Just carry the VLANs via trunks from switch to switch.

Well, that greatly simplifies things, and it sounds like it should work with a single interface if my small switch doesn’t support LAGG.

Thanks a lot to both of you!