Unifi VLANs with new OPNsense install (Can't get internet access)

[Yosh1]

I am transitioning from pfSense to OPNsense, and decided to also update some parts of my network to implement best practices. Unfortunately, now I cannot get internet access for my LAN and VLANs, yet can get it on my "DEBUG" interface...

Here's my network setup:

OPNsense "WAN" (bge3) interface <--> Fiber modem
OPNsense "DEBUG" (bge0) interface <--> PC (This works and provides internet)

For my VLANs (All assigned to ix0 interface, each with static IP and enabled with DHCP):

VLAN 1: "ROUTING" (The idea was to use this for the trunks and routing in the rack, and have LAN as separate VLAN)
VLAN 10: "LAN"
VLAN 99: "NOT"
VLAN 107: "IOT"

OPNsense ix0 interface <--> Unifi Dream Machine Pro (Tagged, Allow All, Native VLAN 1) <--> USW Pro HD 24 (Tagged, Allow All, Native VLAN 1) <--> USW Pro Max 16 PoE (Tagged, Allow All, Native VLAN 1) <--> Unifi AP (Tagged, Allow All, Native VLAN 1)

Since I have two USW switches daisy chained, I was thinking that it has something to do with what I have each port set to in the chain, but tried many permutations and no joy. As example, the connection from the Dream Machine to the 24-port switch is to one of the SFP ports (Tagged, Allow All) and then exits via another SFP port (also set to Tagged, Allow All), then to the SFP port on the 16-port (same - Tagged, Allow All) before the AP port (Tagged, Allow All).

What should the Native VLAN be for each of those steps in the chain? I thought that it would drop packets that enter the trunk if it matches the Native VLAN setting of the trunk port, but setting it to None (what I thought should make it a true trunk) caused no traffic to pass - but setting them all to 1 (not intuitive, but what I have them all set to now) is the closest I've got - at least I can manage all of the switches, but I cannot get internet access on any of the ports (hardwired or WiFi). Yet plugging directly into the back of the OPNsense server on the "DEBUG" interface I created works fine.

I am using Unbound DNS and it's listening on all interfaces.
The WiFi and hardwired connections I am trying are for the LAN VLAN (#10), even with "allow any" rules for VLANs 1 and 10.

Thoughts?

[meyergru]

1. You need to create firewall rules for each but the first (V)LAN (this already has an "allow any -> any" rule).
2. You should not mix tagged and untagged VLANs on the same interface (it causes all kinds of subtle problems). Unifi does this and even prefers it, you did well to separate all VLANs on one pyhsical interface and untagged DEBUG on another.
3. Be careful / do not use VLAN 1: Many manufacturers, including Ubiquiti, use that to denote "untagged". For some, it it only how they handle the untagged (V)LAN internally, others handle VLAN 1 and untagged the same. If you do not want to think about this, simply do not use it.

[nero355]

Some UniFi related posts I have made recently together with some stuff you might also need since you are no longer using an UniFi Router as the Main Router :
- https://forum.opnsense.org/index.php?topic=51014.msg260965#msg260965
- https://forum.opnsense.org/index.php?topic=51054.msg261171#msg261171
- https://forum.opnsense.org/index.php?topic=50960.msg261589#msg261589
- https://forum.opnsense.org/index.php?topic=51099.msg261506#msg261506
- https://forum.opnsense.org/index.php?topic=51099.msg261563#msg261563
- https://forum.opnsense.org/index.php?topic=51118.msg261610#msg261610

Let me know if you need anything else : UniFi Switches + In Wall Accesspoints + OPNsense are running just fine here! :)

[Yosh1]

Ok, thanks for the tips. I tore-down the whole system and rebuilt it - it's better, but it's not working yet.

Now, I can connect to the Unifi APs, and am given a valid IP address from DHCP... but no internet access. I am given the firewall IP (192.168.1.1) for the DNS, which is correct since I'm using Unbound. Again, using the other NIC port setup on my OPNsense box as "DEBUG" with a different IP range works and has internet access.

Everything looks much better in my Unifi Network now (Dream Machine Pro) - none of the devices are dropping in/out and all switches and APs are identified - though I set them to use unique static IPs as they kept falling back to the 192.168.1.20 default which was causing chaos. They didn't appear to get their static ARP addresses from the DHCP server in OPNsense.

I believe that it must still be something with the tagging/trunking on my Unifi chain, if you could take a look - I made a picture to make it easier to follow, this is what I have now:

You cannot view this attachment.

[Yosh1]

Some more data points:

• While I can access the internet and any of the other VLANs through an untagged VLAN 10 port on the 24-port switch, I cannot access the Dream Machine's WebGUI, which has a static IP address of 192.168.1.2 on VLAN 1. VLAN 10 is configured as 192.168.1.1/24, so why can I not reach 192.168.1.2 on VLAN 1?
• The AP on the 16-port switch allows clients to connect, they get the correct IP and DNS depending on the VLAN subnet, but I cannot get internet access on WiFi. Not even by forcing a public DNS with a static IP address - the clients simply cannot get out to the net, even though the hardwired connections in the same switch can.

Is there anything in OPNsense that I can use to better diagnose what's going on? Or is it simply a problem I have with the tagging/VLANs that is a Unifi issue?

[meyergru]

Did you actually create firewall rules (see #1 from my previous post)? Or do you expect that it works without them (hint: it doesn't)?

[mooh]

You need to explain what role the Dream Machine has in your setup. By default it is a network controller, a router, DHCP server and a firewall and lots more. You need to turn off all features except the network controller app or you'll run the risk of services on OPNsense and Dream machine competing against each other.

Also, the OPNsense needs to be connected to one of the internal ports of the dream machine, not one of its WAN ports.

To avoid all of these problems, you may be better off with a Unifi CloudKey or just the Unifi Network Application running on a server instead of the Dream Machine.

Assuming you got that right, let's start with the fact that you didn't mention which VLAN is used for the management network of your unifi devices (the one they use to communicate with the network app). Assuming it is 1, the Unifi default, your OPNsense will never see any traffic on that network as it doesn't have an interface on it. Consequentially, your Unifi Switches and APs never get an answer from DHCP and fall back to their default 192.168.1.20.

In terms of debugging: Turn on logging of all your firewall rules and the default rules to see what your OPNsense sees and drops or allows. And, as mentioned by meyergru, OPNsense follows the philosophy of "deny by default", i.e. no traffic is accepted unless explicitly configured. In contrast, Unifi firewalls are "allow by default" if I remember correctly.

[meyergru]

@mooh is correct abount VLAN 1 (untagged) as being the Unifi management network by default. It can be done differently, but it is hard to do.

From youe first post, I assumed that your DEBUG network is VLAN 1 (untagged) connected to another port on the switch. That is what I meant by "you did well to separate all VLANs on one physical interface and untagged DEBUG on another". Actually, you have it completely separated, which will not work.

Of course, that assumes you actually connect that OpnSense port to an untagged port on your switch. By doing that, you have all tagged ports in OpnSense on one physical port and the untagged port on another, thereby following the rule "do not mix tagged and untagged VLANs on the same port".

[nero355]

Everything looks much better in my Unifi Network now (Dream Machine Pro) - none of the devices are dropping in/out and all switches and APs are identified - though I set them to use unique static IPs as they kept falling back to the 192.168.1.20 default which was causing chaos. They didn't appear to get their static ARP addresses from the DHCP server in OPNsense.

Another TIP :

Make sure you have got both Static IP Addressing and Static DHCP Mappings configured on/for all your Switches and Accesspoints !!

I have done it like this :

- 192.168.1.1 to 192.168.1.9 = Routers & Pi-Hole + Unbound Server.
- 192.168.1.10 to 192.168.1.19 = Core Switches
- 192.168.1.20 to 192.168.1.29 = Client Switches
- 192.168.1.30 to 192.168.1.39 = Indoor Accesspoints
- 192.168.1.40 to 192.168.1.49 = Outdoor Accesspoints

This way you always know what to expect when you connect to them via SSH based on their IP Address :)