Unifi VLANs with new OPNsense install (Can't get internet access)

[bbin]

Unifi's "native" intra-vlan L3 routing for switches is handled on vlan 4040, with the default addressing for that vlan as 10.255.253.0/24.  If no devices exist on that subnet when you enable a native Unifi vlan it will assign whatever device that is handling the routing the address 10.255.253.1.

I had been using opnsense as the gateway for all my vlans, but now I'm working through the process to try and migrate the vlan gateways to the Unifi environment.  opnsense needs to first have a vlan device tagged to vlan 4040 on one of your interfaces and configured with the IP address 10.255.253.1.  When you enable the native VLANs on the Unifi switch the switch will automatically create the interface on the Unifi device with the IP 10.255.253.2.  This becomes the transit interface for L3 routing from the Unifi switch to the opnsense firewall.

There are pros and cons here - the main pro being lower latency for LAN traffic.  The con is that ACLs on the Unifi switch are stateless so you don't get as much visibility and control of traffic between VLANs.  If you have IoT or other less trusted VLANs this might require a hybrid configuration where the gateway for more trusted VLANs like home wireless is the Unifi switch while less trusted like IoT use the opnsense firewall as the gateway to allow for stateful rules to manage traffic.

There are some oddities that I am still working through.  My management interface for the Unifi switches is on vlan 1 (untagged) and I am currently seeing lower latency but extremely slow HTTPS traffic with what looks like state errors coming back from the Internet routing in a weird direction.  kea also isn't properly assigning DHCP addresses; I haven't tried with dnsmasq yet.  The solution seems to be moving the management interface on all Unifi devices (as well as the Unifi OS/Unifi network server) to a tagged VLAN managed by the Unifi switch.  It may also require the use of sloppy states, but I haven't gotten that far yet.

Not sure if anyone else (meyergru?) has a Unifi setup where they could experiment with this design.

[meyergru]

I do not use L3 routing on my switches, even if they had it. I did not even know how Unifi does that. With their consumer-level switches, they do not offer it, also, with smaller networks, I prefer to have all routing controlled by OpnSense itself.

L3 switching is something that IMHO is relevant only for enterprise-grade installations. Everything I depicted here is strictly L2 on the switches.

[Yosh1]

Thanks @meyergru for the help. I made the adjustments as you proposed, but kept them on 1 for the 3rd octet:

• UDM: 192.168.1.2/24
  
  • DHCP disabled
  • DHCP Relay to 192.168.1.1
  • All other options disabled (e.g. no DHCP guarding, no isolation)
• OPNsense "MGMT" (VLAN 1): 192.168.1.1/24
  
  • Separate NIC port
  • Plugged into Unifi switch port: Untagged/Native VLAN 1, None tagged
  • No DHCP server... Is that correct?
• OPNsense "LAN" (VLAN 10): 192.168.1.5/24
  
  • Shared NIC port with other VLANs (But the physical interface is not assigned)
  • Plugged into Unifi switch port: No Untagged/Native, All tagged
  • I gave this a *.5 address so that I could enable a DHCP server on it... Is that correct?
    
    • DHCP server has range 192.168.1.160-192.168.1.250
    • DNS servers and gateway both set to "192.168.1.1"
• ** Disabled the other VLAN interfaces (99 & 107) for now, to simplify debugging
• Unbound DNS:
  
  • Network interfaces: All
  • Listen port 53
• Firewall rule for MGMT:
  
  • "Default allow any rule for MGMT"
  • Interface: MGMT
  • Version: IPv4
  • Protocol: *
  • Source: MGMT net
  • Source Port: *
  • Destination: *
  • Destination Port: *
  • Gateway: *
• Firewall rule for LAN:
  
  • "Default allow any rule for LAN"
  • Interface: LAN
  • Version: IPv4
  • Protocol: *
  • Source: LAN net
  • Source Port: *
  • Destination: *
  • Destination Port: *
  • Gateway: *

As it stands, I still cannot get out to the net from a device connected to VLAN 10 through a Unifi AP. The WiFi network is set to VLAN 10. The path between the switches and the AP is the same as I showed in the image in post #3 (Untagged with VLAN 1 and tagged with all). I connect to the WiFi, get an IP address from the DHCP server (it shows in Leases), the client gets "192.168.1.1" for the DNS server, but I cannot get out to the net.

I enabled logging for the pass any rules and see that there's a duplication of actions - an event shows on the "ix0" interface (the parent interface of all of the VLANs, which is not assigned to anything and is not enabled), which gets a "block" but the same exact event is then passed as the next event, now showing as the MGMT interface. What's going on? See image:


You cannot view this attachment.

[Patrick M. Hausen]

You cannot have the same network - 192.168.1.0/24 - on both MGMT and LAN. Also the gateway that the DHCP server gives to clients must be the OPNsense IP address in the relevant VLAN.

[meyergru]

Right ... and you should enable DHCP on OpnSense for every connected local (V)LAN.

Since we are covering basic networking aspects now: Are you quite certain that OpnSense is right for you? ;-)

Maybe you should start reading here, because that is literally point 1 in that article. By deciding to "keep 192.168.1.x", you deliberately chose to not use a configuration pattern I explained, which by itself guarantees that such things do not happen. You will find that more often than not, there are best practices for a reason.

[Yosh1]

I'd say thanks for the help, as I have found the issue, but you lost some credit when you switched to telling me to not use OPNsense because I was having an odd issue, which is disappointing. Perhaps you were just being cheeky, but everyone has to start somewhere when learning and forums are a means to learn - if I had all of the answers like you then I wouldn't need to be here. I'll now pass on what I learned to others.

I have solved the issues I was having and will write about it here, in case anyone else shows up with similar problems. It turns out there were two issues with the setup, coming from pfSense:

• The main issue was a simple checkbox. In the DHCP server setup, I had set static ARP entries for all of the networking devices (to prevent ARP spoofing) and so that I can use WOL for some of my systems. I had set one for my main desktop PC, which is what I was using for setup. I then checked the innocuous looking box in the DHCP server settings to "Enable Static ARP entries" thinking, sure, that will then enable the ones I defined in the static mappings table. Apparently that checkbox makes it so that only those defined devices can communicate, blocking all others, which is counter-intuitive - but I can replicate my issues by simply toggling that checkbox and the whole network goes down. When it was checked, it's why I was banging my head because my desktop PC had internet access, but no WiFi client did - the WiFi clients weren't in the static ARP entries list. Once I unchecked it, all of the WiFi and VLANs worked as normal and the network came alive and it's been working fine now.
• The best practice I was going for, by moving my private LAN to VLAN 10 is cumbersome with Unifi hardware. While you can change the APs to use a different management network, the switches do not seem to work on anything other than VLAN 1 (hardcoded). Since I had to use VLAN 1 for management to maintain control of my Unifi hardware, I was then working to move my trusted LAN to use VLAN 10, since you're not supposed to have an untagged port which matches one of the broadcast WiFi network VLANs (so I wasn't supposed to have a WiFi network for VLAN 1 and then had the port for the AP be untagged for VLAN 1 for management). That became hectic because I then had parallel paths into OPNsense because I wanted to maintain 192.168.1.1/24 (muscle memory) for my trusted LAN (now VLAN 10), yet provide a path for the management traffic (VLAN 1), keeping the UDM Pro (the network controller) in that same range for continuity (e.g. 192.168.1.2). It turns out that the Unifi hardware carves out a special case for VLAN 1 whereas there's a single note on their VLAN setup along that lines that simply "VLAN 1 is different and doesn't apply here". In short, you absolutely can use VLAN 1 for the native/untagged port of APs that broadcast a VLAN 1 SSID. My guess is that they specifically don't tag VLAN 1 traffic if the SSID uses VLAN 1 so that it gets its tag when it hits the untagged port - then it routes like normal. I removed all of the VLAN 10 assignments, went back to the way I had it with pfSense, and it's all been running fine - with all hardware reachable in the network controller and WiFi clients getting the appropriate IPs.(

With that solved, I now can enjoy not being tied to Netgate and their now closed-source pfSense and using the better Wireguard implementation for linking sites. I updated to the latest 26.1.3, which I can see that there's work being done to improve the UI for firewall rules, which I am looking forward to. Everything else is great, but I do miss the ability to apply colored separators with friendly names (e.g. a green bar with "Allow: DHCP and DNS" before those rules), like I had in pfSense, for organization. I am using the categories + tree view, which is workable, but is not as nice when comparing similar rules across different interfaces. It also seems like sorting when in tree view when looking at all rules can cause issues if you try to sort things, versus the old interface style of having separate tables for each interface.

Anyways, all good now - thanks for the assistance. Cheers.

[nero355]

thanks for the assistance. Cheers.

Just added this : https://forum.opnsense.org/index.php?topic=50632.msg258782#msg258782
To my previous reply : https://forum.opnsense.org/index.php?topic=51127.msg261683#msg261683

IMHO that's all to it : Even with another Router in the game! :)