Messages

I'd say thanks for the help, as I have found the issue, but you lost some credit when you switched to telling me to not use OPNsense because I was having an odd issue, which is disappointing. Perhaps you were just being cheeky, but everyone has to start somewhere when learning and forums are a means to learn - if I had all of the answers like you then I wouldn't need to be here. I'll now pass on what I learned to others.

I have solved the issues I was having and will write about it here, in case anyone else shows up with similar problems. It turns out there were two issues with the setup, coming from pfSense:

• The main issue was a simple checkbox. In the DHCP server setup, I had set static ARP entries for all of the networking devices (to prevent ARP spoofing) and so that I can use WOL for some of my systems. I had set one for my main desktop PC, which is what I was using for setup. I then checked the innocuous looking box in the DHCP server settings to "Enable Static ARP entries" thinking, sure, that will then enable the ones I defined in the static mappings table. Apparently that checkbox makes it so that only those defined devices can communicate, blocking all others, which is counter-intuitive - but I can replicate my issues by simply toggling that checkbox and the whole network goes down. When it was checked, it's why I was banging my head because my desktop PC had internet access, but no WiFi client did - the WiFi clients weren't in the static ARP entries list. Once I unchecked it, all of the WiFi and VLANs worked as normal and the network came alive and it's been working fine now.
• The best practice I was going for, by moving my private LAN to VLAN 10 is cumbersome with Unifi hardware. While you can change the APs to use a different management network, the switches do not seem to work on anything other than VLAN 1 (hardcoded). Since I had to use VLAN 1 for management to maintain control of my Unifi hardware, I was then working to move my trusted LAN to use VLAN 10, since you're not supposed to have an untagged port which matches one of the broadcast WiFi network VLANs (so I wasn't supposed to have a WiFi network for VLAN 1 and then had the port for the AP be untagged for VLAN 1 for management). That became hectic because I then had parallel paths into OPNsense because I wanted to maintain 192.168.1.1/24 (muscle memory) for my trusted LAN (now VLAN 10), yet provide a path for the management traffic (VLAN 1), keeping the UDM Pro (the network controller) in that same range for continuity (e.g. 192.168.1.2). It turns out that the Unifi hardware carves out a special case for VLAN 1 whereas there's a single note on their VLAN setup along that lines that simply "VLAN 1 is different and doesn't apply here". In short, you absolutely can use VLAN 1 for the native/untagged port of APs that broadcast a VLAN 1 SSID. My guess is that they specifically don't tag VLAN 1 traffic if the SSID uses VLAN 1 so that it gets its tag when it hits the untagged port - then it routes like normal. I removed all of the VLAN 10 assignments, went back to the way I had it with pfSense, and it's all been running fine - with all hardware reachable in the network controller and WiFi clients getting the appropriate IPs.(

With that solved, I now can enjoy not being tied to Netgate and their now closed-source pfSense and using the better Wireguard implementation for linking sites. I updated to the latest 26.1.3, which I can see that there's work being done to improve the UI for firewall rules, which I am looking forward to. Everything else is great, but I do miss the ability to apply colored separators with friendly names (e.g. a green bar with "Allow: DHCP and DNS" before those rules), like I had in pfSense, for organization. I am using the categories + tree view, which is workable, but is not as nice when comparing similar rules across different interfaces. It also seems like sorting when in tree view when looking at all rules can cause issues if you try to sort things, versus the old interface style of having separate tables for each interface.

Anyways, all good now - thanks for the assistance. Cheers.

Thanks @meyergru for the help. I made the adjustments as you proposed, but kept them on 1 for the 3rd octet:

• UDM: 192.168.1.2/24
  
  • DHCP disabled
  • DHCP Relay to 192.168.1.1
  • All other options disabled (e.g. no DHCP guarding, no isolation)
• OPNsense "MGMT" (VLAN 1): 192.168.1.1/24
  
  • Separate NIC port
  • Plugged into Unifi switch port: Untagged/Native VLAN 1, None tagged
  • No DHCP server... Is that correct?
• OPNsense "LAN" (VLAN 10): 192.168.1.5/24
  
  • Shared NIC port with other VLANs (But the physical interface is not assigned)
  • Plugged into Unifi switch port: No Untagged/Native, All tagged
  • I gave this a *.5 address so that I could enable a DHCP server on it... Is that correct?
    
    • DHCP server has range 192.168.1.160-192.168.1.250
    • DNS servers and gateway both set to "192.168.1.1"
• ** Disabled the other VLAN interfaces (99 & 107) for now, to simplify debugging
• Unbound DNS:
  
  • Network interfaces: All
  • Listen port 53
• Firewall rule for MGMT:
  
  • "Default allow any rule for MGMT"
  • Interface: MGMT
  • Version: IPv4
  • Protocol: *
  • Source: MGMT net
  • Source Port: *
  • Destination: *
  • Destination Port: *
  • Gateway: *
• Firewall rule for LAN:
  
  • "Default allow any rule for LAN"
  • Interface: LAN
  • Version: IPv4
  • Protocol: *
  • Source: LAN net
  • Source Port: *
  • Destination: *
  • Destination Port: *
  • Gateway: *

As it stands, I still cannot get out to the net from a device connected to VLAN 10 through a Unifi AP. The WiFi network is set to VLAN 10. The path between the switches and the AP is the same as I showed in the image in post #3 (Untagged with VLAN 1 and tagged with all). I connect to the WiFi, get an IP address from the DHCP server (it shows in Leases), the client gets "192.168.1.1" for the DNS server, but I cannot get out to the net.

I enabled logging for the pass any rules and see that there's a duplication of actions - an event shows on the "ix0" interface (the parent interface of all of the VLANs, which is not assigned to anything and is not enabled), which gets a "block" but the same exact event is then passed as the next event, now showing as the MGMT interface. What's going on? See image:


You cannot view this attachment.

I think you misread my "DEBUG" network - it's 192.168.3.1, not 192.168.1.3. I use it to prevent myself from being locked-out when working with the 192.168.1.1 range, as it's a separate NIC with it's own DHCP server.

I am now looking at setting it up as you suggest. What IPs/parameters would I set for the following:

• "Management" VLAN 1 setup in OPNsense. Has it's own NIC port on OPNsense and is tied to an untagged/native VLAN 1 port in the Unifi switch.
• UDM Pro setup. Right now I have the "Dream Machine Router" set to 192.168.1.2, with DHCP and other services turned off. "3rd Party Gateway" entries for each of the VLANs (#'s 10, 99, 107)
• "LAN" VLAN 10 in OPNsense. Would like to access the VLAN 1 devices, if possible (connect to UDM Pro to see network controller)
• All other VLANs in OPNsense. Currently have them all set as VLANs on top of ix0 interface (separate NIC), which is tied to an Allow All tagged and no untagged/native in the Unifi switch

Any specific DNS or firewall rules that I should be aware of? I currently just have Allow Any rules for MGMT (VLAN 1) and LAN (VLAN 10) networks.

For my layout, I like my trusted devices ("LAN") to be in the 192.168.1.1-255 range - just muscle memory. My existing network uses that range, with many uses based on existing static IP addresses that would be a lot of work to change now.
For my VLANs, I set the 3rd octet to match the VLAN ID (e.g. 192.168.107.1 for the "IOT" VLAN 107).

To fix the VLAN 1 issue, I now have OPNsense setup with a trunk to it (no native, only Allow All for tagged) that is intended to carry all of the VLANs other than 1. I am now setting up the untagged VLAN 1 network:
When I setup OPNsense, I set the "lan" (what OPNsense refers to it as) on a spare NIC (bge1). WAN is on bge0. I didn't have that going anywhere, so I have now enabled it in OPNsense as "MGMT" (Management), but I don't know what to do from there. I figure that it will need a static IP address assigned? But then how do I prevent it from interfering with my "LAN" (VLAN 10) network, which has the gateway at 192.168.1.1. It's just needed for routing the management network (e.g. switch-to-switch, APs to switch), so I figure that it will need a firewall rule to allow any? But if I set a static IP (say 192.168.1.5/32), how do I make the Unifi devices use it? Set them to use a gateway of 192.168.1.5?

Everything is turned off on the Dream Machine. I just use it as a controller for the Unifi devices, as I like its form-factor. The standard LAN is set to 192.168.1/24, with it's own IP set as a static 192.168.1.2.

Some more data points:

• While I can access the internet and any of the other VLANs through an untagged VLAN 10 port on the 24-port switch, I cannot access the Dream Machine's WebGUI, which has a static IP address of 192.168.1.2 on VLAN 1. VLAN 10 is configured as 192.168.1.1/24, so why can I not reach 192.168.1.2 on VLAN 1?
• The AP on the 16-port switch allows clients to connect, they get the correct IP and DNS depending on the VLAN subnet, but I cannot get internet access on WiFi. Not even by forcing a public DNS with a static IP address - the clients simply cannot get out to the net, even though the hardwired connections in the same switch can.

Is there anything in OPNsense that I can use to better diagnose what's going on? Or is it simply a problem I have with the tagging/VLANs that is a Unifi issue?

Ok, thanks for the tips. I tore-down the whole system and rebuilt it - it's better, but it's not working yet.

Now, I can connect to the Unifi APs, and am given a valid IP address from DHCP... but no internet access. I am given the firewall IP (192.168.1.1) for the DNS, which is correct since I'm using Unbound. Again, using the other NIC port setup on my OPNsense box as "DEBUG" with a different IP range works and has internet access.

Everything looks much better in my Unifi Network now (Dream Machine Pro) - none of the devices are dropping in/out and all switches and APs are identified - though I set them to use unique static IPs as they kept falling back to the 192.168.1.20 default which was causing chaos. They didn't appear to get their static ARP addresses from the DHCP server in OPNsense.

I believe that it must still be something with the tagging/trunking on my Unifi chain, if you could take a look - I made a picture to make it easier to follow, this is what I have now:

You cannot view this attachment.

I am transitioning from pfSense to OPNsense, and decided to also update some parts of my network to implement best practices. Unfortunately, now I cannot get internet access for my LAN and VLANs, yet can get it on my "DEBUG" interface...

Here's my network setup:

OPNsense "WAN" (bge3) interface <--> Fiber modem
OPNsense "DEBUG" (bge0) interface <--> PC (This works and provides internet)

For my VLANs (All assigned to ix0 interface, each with static IP and enabled with DHCP):

VLAN 1: "ROUTING" (The idea was to use this for the trunks and routing in the rack, and have LAN as separate VLAN)
VLAN 10: "LAN"
VLAN 99: "NOT"
VLAN 107: "IOT"

OPNsense ix0 interface <--> Unifi Dream Machine Pro (Tagged, Allow All, Native VLAN 1) <--> USW Pro HD 24 (Tagged, Allow All, Native VLAN 1) <--> USW Pro Max 16 PoE (Tagged, Allow All, Native VLAN 1) <--> Unifi AP (Tagged, Allow All, Native VLAN 1)

Since I have two USW switches daisy chained, I was thinking that it has something to do with what I have each port set to in the chain, but tried many permutations and no joy. As example, the connection from the Dream Machine to the 24-port switch is to one of the SFP ports (Tagged, Allow All) and then exits via another SFP port (also set to Tagged, Allow All), then to the SFP port on the 16-port (same - Tagged, Allow All) before the AP port (Tagged, Allow All).

What should the Native VLAN be for each of those steps in the chain? I thought that it would drop packets that enter the trunk if it matches the Native VLAN setting of the trunk port, but setting it to None (what I thought should make it a true trunk) caused no traffic to pass - but setting them all to 1 (not intuitive, but what I have them all set to now) is the closest I've got - at least I can manage all of the switches, but I cannot get internet access on any of the ports (hardwired or WiFi). Yet plugging directly into the back of the OPNsense server on the "DEBUG" interface I created works fine.

I am using Unbound DNS and it's listening on all interfaces.
The WiFi and hardwired connections I am trying are for the LAN VLAN (#10), even with "allow any" rules for VLANs 1 and 10.

Thoughts?