Unifi VLANs with new OPNsense install (Can't get internet access)

[Yosh1]

I am transitioning from pfSense to OPNsense, and decided to also update some parts of my network to implement best practices. Unfortunately, now I cannot get internet access for my LAN and VLANs, yet can get it on my "DEBUG" interface...

Here's my network setup:

OPNsense "WAN" (bge3) interface <--> Fiber modem
OPNsense "DEBUG" (bge0) interface <--> PC (This works and provides internet)

For my VLANs (All assigned to ix0 interface, each with static IP and enabled with DHCP):

VLAN 1: "ROUTING" (The idea was to use this for the trunks and routing in the rack, and have LAN as separate VLAN)
VLAN 10: "LAN"
VLAN 99: "NOT"
VLAN 107: "IOT"

OPNsense ix0 interface <--> Unifi Dream Machine Pro (Tagged, Allow All, Native VLAN 1) <--> USW Pro HD 24 (Tagged, Allow All, Native VLAN 1) <--> USW Pro Max 16 PoE (Tagged, Allow All, Native VLAN 1) <--> Unifi AP (Tagged, Allow All, Native VLAN 1)

Since I have two USW switches daisy chained, I was thinking that it has something to do with what I have each port set to in the chain, but tried many permutations and no joy. As example, the connection from the Dream Machine to the 24-port switch is to one of the SFP ports (Tagged, Allow All) and then exits via another SFP port (also set to Tagged, Allow All), then to the SFP port on the 16-port (same - Tagged, Allow All) before the AP port (Tagged, Allow All).

What should the Native VLAN be for each of those steps in the chain? I thought that it would drop packets that enter the trunk if it matches the Native VLAN setting of the trunk port, but setting it to None (what I thought should make it a true trunk) caused no traffic to pass - but setting them all to 1 (not intuitive, but what I have them all set to now) is the closest I've got - at least I can manage all of the switches, but I cannot get internet access on any of the ports (hardwired or WiFi). Yet plugging directly into the back of the OPNsense server on the "DEBUG" interface I created works fine.

I am using Unbound DNS and it's listening on all interfaces.
The WiFi and hardwired connections I am trying are for the LAN VLAN (#10), even with "allow any" rules for VLANs 1 and 10.

Thoughts?

[meyergru]

1. You need to create firewall rules for each but the first (V)LAN (this already has an "allow any -> any" rule).
2. You should not mix tagged and untagged VLANs on the same interface (it causes all kinds of subtle problems). Unifi does this and even prefers it, you did well to separate all VLANs on one pyhsical interface and untagged DEBUG on another.
3. Be careful / do not use VLAN 1: Many manufacturers, including Ubiquiti, use that to denote "untagged". For some, it it only how they handle the untagged (V)LAN internally, others handle VLAN 1 and untagged the same. If you do not want to think about this, simply do not use it.

[nero355]

Some UniFi related posts I have made recently together with some stuff you might also need since you are no longer using an UniFi Router as the Main Router :
- https://forum.opnsense.org/index.php?topic=51014.msg260965#msg260965
- https://forum.opnsense.org/index.php?topic=51054.msg261171#msg261171
- https://forum.opnsense.org/index.php?topic=50960.msg261589#msg261589
- https://forum.opnsense.org/index.php?topic=51099.msg261506#msg261506
- https://forum.opnsense.org/index.php?topic=51099.msg261563#msg261563
- https://forum.opnsense.org/index.php?topic=51118.msg261610#msg261610

Let me know if you need anything else : UniFi Switches + In Wall Accesspoints + OPNsense are running just fine here! :)

[Yosh1]

Ok, thanks for the tips. I tore-down the whole system and rebuilt it - it's better, but it's not working yet.

Now, I can connect to the Unifi APs, and am given a valid IP address from DHCP... but no internet access. I am given the firewall IP (192.168.1.1) for the DNS, which is correct since I'm using Unbound. Again, using the other NIC port setup on my OPNsense box as "DEBUG" with a different IP range works and has internet access.

Everything looks much better in my Unifi Network now (Dream Machine Pro) - none of the devices are dropping in/out and all switches and APs are identified - though I set them to use unique static IPs as they kept falling back to the 192.168.1.20 default which was causing chaos. They didn't appear to get their static ARP addresses from the DHCP server in OPNsense.

I believe that it must still be something with the tagging/trunking on my Unifi chain, if you could take a look - I made a picture to make it easier to follow, this is what I have now:

You cannot view this attachment.

[Yosh1]

Some more data points:

• While I can access the internet and any of the other VLANs through an untagged VLAN 10 port on the 24-port switch, I cannot access the Dream Machine's WebGUI, which has a static IP address of 192.168.1.2 on VLAN 1. VLAN 10 is configured as 192.168.1.1/24, so why can I not reach 192.168.1.2 on VLAN 1?
• The AP on the 16-port switch allows clients to connect, they get the correct IP and DNS depending on the VLAN subnet, but I cannot get internet access on WiFi. Not even by forcing a public DNS with a static IP address - the clients simply cannot get out to the net, even though the hardwired connections in the same switch can.

Is there anything in OPNsense that I can use to better diagnose what's going on? Or is it simply a problem I have with the tagging/VLANs that is a Unifi issue?

[meyergru]

Did you actually create firewall rules (see #1 from my previous post)? Or do you expect that it works without them (hint: it doesn't)?

[mooh]

You need to explain what role the Dream Machine has in your setup. By default it is a network controller, a router, DHCP server and a firewall and lots more. You need to turn off all features except the network controller app or you'll run the risk of services on OPNsense and Dream machine competing against each other.

Also, the OPNsense needs to be connected to one of the internal ports of the dream machine, not one of its WAN ports.

To avoid all of these problems, you may be better off with a Unifi CloudKey or just the Unifi Network Application running on a server instead of the Dream Machine.

Assuming you got that right, let's start with the fact that you didn't mention which VLAN is used for the management network of your unifi devices (the one they use to communicate with the network app). Assuming it is 1, the Unifi default, your OPNsense will never see any traffic on that network as it doesn't have an interface on it. Consequentially, your Unifi Switches and APs never get an answer from DHCP and fall back to their default 192.168.1.20.

In terms of debugging: Turn on logging of all your firewall rules and the default rules to see what your OPNsense sees and drops or allows. And, as mentioned by meyergru, OPNsense follows the philosophy of "deny by default", i.e. no traffic is accepted unless explicitly configured. In contrast, Unifi firewalls are "allow by default" if I remember correctly.

[meyergru]

@mooh is correct abount VLAN 1 (untagged) as being the Unifi management network by default. It can be done differently, but it is hard to do.

From your first post, I assumed that your DEBUG network is VLAN 1 (untagged) connected to another port on the switch. That is what I meant by "you did well to separate all VLANs on one physical interface and untagged DEBUG on another". Actually, you have the management network completely separated, which will most probably not work.

Of course, that assumes you actually connect that OpnSense port to an untagged port on your switch. By doing that, you have all tagged ports in OpnSense on one physical port and the untagged port on another, thereby following the rule "do not mix tagged and untagged VLANs on the same port".

[nero355]

Everything looks much better in my Unifi Network now (Dream Machine Pro) - none of the devices are dropping in/out and all switches and APs are identified - though I set them to use unique static IPs as they kept falling back to the 192.168.1.20 default which was causing chaos. They didn't appear to get their static ARP addresses from the DHCP server in OPNsense.

Another TIP :

Make sure you have got both Static IP Addressing and Static DHCP Mappings configured on/for all your Switches and Accesspoints !!

I have done it like this :

- 192.168.1.1 to 192.168.1.9 = Routers & Pi-Hole + Unbound Server.
- 192.168.1.10 to 192.168.1.19 = Core Switches
- 192.168.1.20 to 192.168.1.29 = Client Switches
- 192.168.1.30 to 192.168.1.39 = Indoor Accesspoints
- 192.168.1.40 to 192.168.1.49 = Outdoor Accesspoints

This way you always know what to expect when you connect to them via SSH based on their IP Address :)

[meyergru]

In fact, I would never use that subnet at all for the reasons explained here.

[nero355]

In fact, I would never use that subnet at all for the reasons explained here.

And you are right! :)


But...


- Ubiquiti UniFi devices and anything outside that range can be a mess when something goes wrong :(
- The chance of me EVER needing/using a VPN is pretty much ZERO.
- I am well aware of VPN IPv4 Address Range conflicts for at least 10 to 15 years now so if I absolutely need to change something at some point then so be it...

[Yosh1]

For my layout, I like my trusted devices ("LAN") to be in the 192.168.1.1-255 range - just muscle memory. My existing network uses that range, with many uses based on existing static IP addresses that would be a lot of work to change now.
For my VLANs, I set the 3rd octet to match the VLAN ID (e.g. 192.168.107.1 for the "IOT" VLAN 107).

To fix the VLAN 1 issue, I now have OPNsense setup with a trunk to it (no native, only Allow All for tagged) that is intended to carry all of the VLANs other than 1. I am now setting up the untagged VLAN 1 network:
When I setup OPNsense, I set the "lan" (what OPNsense refers to it as) on a spare NIC (bge1). WAN is on bge0. I didn't have that going anywhere, so I have now enabled it in OPNsense as "MGMT" (Management), but I don't know what to do from there. I figure that it will need a static IP address assigned? But then how do I prevent it from interfering with my "LAN" (VLAN 10) network, which has the gateway at 192.168.1.1. It's just needed for routing the management network (e.g. switch-to-switch, APs to switch), so I figure that it will need a firewall rule to allow any? But if I set a static IP (say 192.168.1.5/32), how do I make the Unifi devices use it? Set them to use a gateway of 192.168.1.5?

Everything is turned off on the Dream Machine. I just use it as a controller for the Unifi devices, as I like its form-factor. The standard LAN is set to 192.168.1/24, with it's own IP set as a static 192.168.1.2.

[meyergru]

No, I meant DEBUG is what most people call "management network", not to connect WAN to your switch.

To be exakt: Unifi wants its management network to be "VLAN 1" (= untagged), and you can define any user networks as VLANs, yet OpnSense does not like mixing tagged and untagged networks on the same NIC. So, you could use your DEBUG network of 192.168.3.1/24 as management network on VLAN 1 (untagged) with one separate physical of OpnSense connected to that via the switch and another NIC with all of the "real" (= tagged) VLANs. WAN is still on a separate NIC connected to your OpnSense only.

By doing this you avoid using 192.168.1.0/24 altogether, BTW. This might come in handy once you find that your DSL modem or ONT uses 192.168.1.1/24. If you did use that for your management network (or any other (V)LAN, you would have a hard time to access the ONT web interface on another OpnSense NIC.

And sure, you turn DHCP off on your dream machine and use DHCP only from your OpnSense. This will also assign all Unifi devices their IPs, except maybe for your dream machine, which presumably has a static IP that you must assign yourself (like 192.168.3.2/24 in this example).

[Yosh1]

I think you misread my "DEBUG" network - it's 192.168.3.1, not 192.168.1.3. I use it to prevent myself from being locked-out when working with the 192.168.1.1 range, as it's a separate NIC with it's own DHCP server.

I am now looking at setting it up as you suggest. What IPs/parameters would I set for the following:

• "Management" VLAN 1 setup in OPNsense. Has it's own NIC port on OPNsense and is tied to an untagged/native VLAN 1 port in the Unifi switch.
• UDM Pro setup. Right now I have the "Dream Machine Router" set to 192.168.1.2, with DHCP and other services turned off. "3rd Party Gateway" entries for each of the VLANs (#'s 10, 99, 107)
• "LAN" VLAN 10 in OPNsense. Would like to access the VLAN 1 devices, if possible (connect to UDM Pro to see network controller)
• All other VLANs in OPNsense. Currently have them all set as VLANs on top of ix0 interface (separate NIC), which is tied to an Allow All tagged and no untagged/native in the Unifi switch

Any specific DNS or firewall rules that I should be aware of? I currently just have Allow Any rules for MGMT (VLAN 1) and LAN (VLAN 10) networks.

[meyergru]

I had a typo, of course I meant 192.168.3.1/24. That would be the management network on VLAN 1 (untagged).

The UDM Pro would also be on VLAN 1 with 192.168.3.2/24, gateway = 192.168.3.1 (OpnSense). The other VLANs can be defined, but the UDM Pro does not enable DHCP for any of those. Thus, the port it connects to on the switch must be a trunk with untagged VLAN 1 and all VLANs. That is just because the UDM Pro still ist the network controller for the APs and switches.

I like to have the third octet of the network be set the same as the VLAN, so VLAN 10 would be 192.168.10.0/24 a.s.o. The management network is an exemption, because it has VLAN 1 (= untagged).

The OpnSense has two legs, one untagged on one port and one as trunk with all the other VLANs. You can connect that second leg as trunk to the switch as well (it is only that the "parent" NIC for all the VLANs will not be configured on OpnSense.

You will have to define a PASS ALL rule for all VLANs (including MANAGEMENT) to "any" first, to see all of those get internet access.
Of course, this will enable to access any other VLAN, first. Then, you can put block rules before the "PASS ALL" rule to inhibit access from any VLAN as you wish. Usually, that would mean than you LAN (VLAN 10) will either not have any block rule at all or only to the management VLAN.
You can also define specific MAC or IP aliases on your LAN to pass to the MANAGEMENT VLAN.

For most other VLANs, you will most probably want a blocking rule including the "VLAN net"s (there are pre-defined network range aliases for any OpnSense network interface) of all other local VLANs or even a self-created alias including all RFC1918 networks. This is safe, because local traffic in any local VLAN does not pass OpnSense, so it does not block traffic for itself.

Of course, that RFC1918 block rule would also block access to OpnSense services, like DNS, NTP and others. Thus, you will have to specify rules to allow those types of traffic even before your "block RFC1918" rule.

Typically, what you will end up with is a set of rules like these on each interface:

1. Allow specific traffic to OpnSense services (like NTP or DNS).
2. Allow specific clients to do "anything they want" (i.e. your management clients).
3. Block access to specific local networks (like the management network or all other VLANs or even RFC1918).
4. Allow to "any" (i.e. internet access).

How you do that is up to you, however, when you first start out with the "ALLOW ANY" rules as depicted, everything should work and you can set limits from there. Note, that you will find many discussions here where people fight over the pros and cons of one way or the other to do it (e.g. "block RFC1918 rule before allow any" or "allow only to !RFC1918.", combining #3 and #4 above). Even "RFC1918" as an alias is debatable, see: https://forum.opnsense.org/index.php?topic=46094.0