This is about a brand-new Unifi USW-Pro-HD-24-POE switch. I though I write this here to save others the time to find out themselves:
I recently bought one of those switches and wanted to use the LAGG feature with my OpnSense box with 4 I226V NICs.
My previous setup was to use two of those interfaces, one carrying my main (V)LAN and the other to carry all other VLANs, e.g. IoT. With that, I could have cross-VLAN traffic at 2.5 Gbps in both directions.
So I tried setting up a LAGG with LACP layer 4, in order to have TCP ports come into the mix for hashing and distributing different TCP streams over two physical links. I tested with "iperf3 -c x.x.x.x -P4" between two Linux VM hosts on different VLANs to have multiple source ports and to my great disbelief, got only 2.5 Gbps. The same thing happened when I tried against OpnSense itself.
I also tried two different iperf3 instances from the same machine to no avail.
After some more trial and error, I found that the Unifi switch can only handle layer 2 & 3 hashing, which is especially disappointing when you know that they did better years ago (https://community.ui.com/questions/LACP-connection-proof-of-evidence/849b7056-255d-41df-a633-125faaa43909). Only when I had two different clients do independent runs, could I have > 4 Gbps in sum - and even this took a few tries before I found a counterpart machine with a matching MAC.
I then found this tidbit in the CLI of the switch:
BusyBox v1.25.1 () built-in shell (ash)
___ ___ .__________.__
| | |____ |__\_ ____/__|
| | / \| || __) | | (c) 2010-2024
| | | | \ || \ | | Ubiquiti Inc.
|______|___| /__||__/ |__|
|_/ https://www.ui.com
Welcome to UniFi USW-Pro-HD-24-PoE!
********************************* NOTICE **********************************
* By logging in to, accessing, or using any Ubiquiti product, you are *
* signifying that you have read our Terms of Service (ToS) and End User *
* License Agreement (EULA), understand their terms, and agree to be *
* fully bound to them. The use of SSH (Secure Shell) can potentially *
* harm Ubiquiti devices and result in lost access to them and their data. *
* By proceeding, you acknowledge that the use of SSH to modify device(s) *
* outside of their normal operational scope, or in any manner *
* inconsistent with the ToS or EULA, will permanently and irrevocably *
* void any applicable warranty. *
***************************************************************************
hagen-US3.7.1.33# cli
hagen# configure
hagen(config)# lag load-balance
Incomplete command
hagen(config)# lag load-balance ?
src-dst-mac LAG load balancing is based on source and destination MAC addr
ess.
src-dst-mac-ip LAG load balancing is based on source and destination of MAC a
nd IP addresses.
hagen(config)#
This shows that only layer 2 & 3 are being supported, as my tests confirmed.
Actually, for my scenario, LAGGs are worse without layer 4 hashing than splitting the VLANs across the interfaces, as I would have to be lucky (chances are 50:50) to have two arbitrary devices be put on different links via layer 2 or 3 hashing, even when I use multiple TCP streams with different ports.
I also posted this to the Unifi forum (https://community.ui.com/questions/Do-Unifi-switches-support-only-layer-2-and-3-LACP-distribution-algorithms/93eff56f-77b3-458f-8500-55822c276fc8).
Oh, BTW: Today, I looked at my USW-Enterprise-24-PoE. It has a completely different CLI structure and obviously, this one does in fact support LACP layer 4:
BusyBox v1.25.1 () built-in shell (ash)
___ ___ .__________.__
| | |____ |__\_ ____/__|
| | / \| || __) | | (c) 2010-2024
| | | | \ || \ | | Ubiquiti Inc.
|______|___| /__||__/ |__|
|_/ https://www.ui.com
Welcome to UniFi USW-Enterprise-24-PoE!
********************************* NOTICE **********************************
* By logging in to, accessing, or using any Ubiquiti product, you are *
* signifying that you have read our Terms of Service (ToS) and End User *
* License Agreement (EULA), understand their terms, and agree to be *
* fully bound to them. The use of SSH (Secure Shell) can potentially *
* harm Ubiquiti devices and result in lost access to them and their data. *
* By proceeding, you acknowledge that the use of SSH to modify device(s) *
* outside of their normal operational scope, or in any manner *
* inconsistent with the ToS or EULA, will permanently and irrevocably *
* void any applicable warranty. *
***************************************************************************
edgar-US.7.1.26# cli
Entering character mode
Escape character is '^]'.
Warning!
The changes may break controller settings and only be effective until reboot.
(UBNT) >enable
(UBNT) #configure
(UBNT) (Config)#port-channel load-balance ?
1 Src MAC, VLAN, EType, incoming port
2 Dest MAC, VLAN, EType, incoming port
3 Src/Dest MAC, VLAN, EType, incoming port
4 Src IP and Src TCP/UDP Port fields
5 Dest IP and Dest TCP/UDP Port fields
6 Src/Dest IP and TCP/UDP Port fields
(UBNT) (Config)#port-channel load-balance
So, it seems model-dependend, probably based on product lines (i.e. Pro vs. Enterprise).
Well that's a bit disappointing,
I would expect on the PRO tier to support such features. But looks like its missing.
Even tho you tested and proofed it yourself, did you contact the Vendor asking them if they will in the future add this feature with future updates?
Regards,
S.
I do not think they will change that, because from the looks of it, it may well be that the problem lies with the chipset firmware, so they probably have to wait for a fix from RealTek. You can see that when you get into the CLI level. The whole menu structure is different from the Broadcom-based Enterprise line of Unifi switches.
And they might argue that formally, LACP does not call for any specific distribution method. After all, even including port numbers would not help with a single TCP stream. Bonding is simply not the same as having more bandwidth over one link.
However, I have escalated another problem that is way worse than this (and a security problem, as well):
https://community.ui.com/questions/Bug-802-1x-port-security-does-not-work-for-IPv6-on-USW-Pro-HD-24-POE/3280a2d8-43ff-49da-88e0-445af74e203e
Basically, when you use MAC-based 802.1x port security on those switches, your clients devices will not get a VLAN assigned, but instead ALL VLANs are visible on the port. This results in broken IPv6 (since you get prefixes from all VLANs), but you can also see packets from VLANs you should not be able to access.
The case has been escalated to expert level, they will contact me as soon as they know something new.
One more for the database. Looks like the same situation in the "Pro Max" line.
BusyBox v1.25.1 () built-in shell (ash)
___ ___ .__________.__
| | |____ |__\_ ____/__|
| | / \| || __) | | (c) 2010-2024
| | | | \ || \ | | Ubiquiti Inc.
|______|___| /__||__/ |__|
|_/ https://www.ui.com
Welcome to UniFi USW-Pro-Max-16-PoE!
********************************* NOTICE **********************************
* By logging in to, accessing, or using any Ubiquiti product, you are *
* signifying that you have read our Terms of Service (ToS) and End User *
* License Agreement (EULA), understand their terms, and agree to be *
* fully bound to them. The use of SSH (Secure Shell) can potentially *
* harm Ubiquiti devices and result in lost access to them and their data. *
* By proceeding, you acknowledge that the use of SSH to modify device(s) *
* outside of their normal operational scope, or in any manner *
* inconsistent with the ToS or EULA, will permanently and irrevocably *
* void any applicable warranty. *
***************************************************************************
USWProMax16PoE-US2.7.1.26# cli
USWProMax16PoE# configure
USWProMax16PoE(config)# lag load-balance
src-dst-mac LAG load balancing is based on source and destination MAC addr
ess.
src-dst-mac-ip LAG load balancing is based on source and destination of MAC a
nd IP addresses.
...as of f/w version 7.1.26.15869.
Then, probably, those are based on RealTek chipsets, as well. It would be interesting to know if the 802.1x problem exists on that product line, too.
IMHO one must concede to the vendors that layer 4 hashing is not 802.3ad compliant because of the potential for reordered TCP fragments. I am perfectly fine with any equipment supporting 2+3 only (and documenting that fact).
Best explanation I know is this Linux kernel documentation:
https://www.kernel.org/doc/Documentation/networking/bonding.txt
The 802.1x issue you found is severe, though! 🤯 Keep up the pressure.
Quote from: meyergru on February 12, 2025, 10:58:33 AMIt would be interesting to know if the 802.1x problem exists on that product line, as well.
I don't use 802.1x at home but I had exactly the same symptom when I first set up my OPNsense+UniFi network a few months back. My Windows box was getting IPv6 addresses on all VLANs. The issue was due to tags being present on the port that Windows was using (found the solution here: https://forum.turris.cz/t/ipv6-slaac-addresses-for-all-lan-interfaces-configured-on-host-ra-received-for-all-interfaces/18964/3).
Since then I have changed all my access ports to be untagged (and specify the native VLAN for each in UniFi Network) and the issue resolved.
Right, but with TCP window scaling and modern out-of-order processing of current TCP stacks, that is not a problem for the endpoints of the TCP conversation, but merely for very strict 802.3ad implementations. You would find out the moment you connect your links to such (and OpnSense can handle it!).
Quote from: OPNenthu on February 12, 2025, 12:04:43 PMI don't use 802.1x at home but I had exactly the same symptom when I first set up my OPNsense+UniFi network a few months back. My Windows box was getting IPv6 addresses on all VLANs. The issue was due to tags being present on the port that Windows was using (found the solution here: https://forum.turris.cz/t/ipv6-slaac-addresses-for-all-lan-interfaces-configured-on-host-ra-received-for-all-interfaces/18964/3).
Since then I have changed all my access ports to be untagged (and specify the native VLAN for each in UniFi Network) and the issue resolved.
Can you tell us what you use? OpnSense, Unifi APs, connected by what?
I have Unifi infrastructure as well and apart from the 802.1x problem on that specific switch, I had no such problems. The APs are fine.
BTW: Recently, I found out that you can also have one WLAN/SSID with multiple VLANs with 802.1x (https://help.ui.com/hc/en-us/articles/115004589707-MAC-Based-VLAN-Assignment-Using-802-1x-in-UniFi-Network) instead of the usual "one VLAN per WLAN/SSID" (http://with%20802.1x). The idea is to have less SSIDs, only differentiating for different WLAN features.
Both ways, the AP will be on a trunk port and selects the VLAN it passes to the WLAN client, based on either the SSID or via 802.1x based on its MAC. I verified that to work.
It's still evolving as I learn but as of now I have OPNsense on a 4-port Protectli (I226-V) with a 2-port LAGG trunk carrying all the VLANs to/from the USW Pro Max. I will eventually buy a dedicated AP but for now I am using my old Asus WiFi router which has been converted with custom firmware into an AP (routing functions disabled).
The "AP" is connected to the USW Pro Max with a single 1GbE link. It has 4 "LAN" ports which are bridge interfaces (br0 through br3). I have br0 configured with an IP address on VLAN 1 (the UniFi "Default" network) and is the management interface to the AP. The other bridges are serving my "Home", "Guest", and "IoT" subnets respectively with separate SSIDs on the respective VLANs.
In the screenshot ports 15 & 16 (counting left to right) are the LAGG trunk to OPNsense and are aggregated in UniFi Network. Port 14 is my Proxmox trunk. Port 2 is my AP trunk. The AP requires the management VLAN to be untagged, and I've done the same for Proxmox, but for OPNsense I am passing all as tagged only. It took me forever to understand this about OPNsense/FreeBSD.
unifi.png
wifi-router-as-ap.png
The only problem that I'm still facing is that my SLAAC temporary addresses fail to renew once they get deprecated. That old thread of mine is not actually resolved, sadly, and I'm really starting to blame UniFi for it... although I did see some threads here recently about broken IPv6 ND in FreeBSD upstream. I don't know, but I'm frustrated over it.
That's an interesting prospect- to assign the VLAN by client MAC address over 802.1x with RADIUS. I don't know if my AP and all of our clients in the house are capable, but it would solve the problem I currently have of being limited to just a few SSIDs due to the bridge interfaces. You gave me a new project idea. But I have to be careful not to make things inconvenient for the rest of the family with too many enterprise-like restrictions and requirements. I'm only now starting to get back on good terms with my wife after so many internet outages :)
That is a similar setup to mine. I also use trunk ports for Proxmox, APs and OpnSense. I also have two ports for OpnSense, but I do not LAGG them, but as explained, split up the VLANs between those. That way, I can get the full port bandwidth even for one inter-VLAN IP stream, because the traffic passes out another port that it enters. Intra-VLAN traffic does not pass OpnSense anyway, so I am better off than with LAGG.
The VLAN assignment on WiFi is something the APs manage, so you would need Unifi APs for that.
The SLAAC problems you are referring to are mentioned in this thread (https://forum.opnsense.org/index.php?topic=44435), which I already replied to. I have no problems with those.
As I said, if you see multiple VLANs at once, IPv6 will definitely show problems, because then, multiple prefixes will get announced via RA. The situation is less problematic with IPv4, because when you use static reservations, they will get assigned regardless of the VLAN they are being sent over. Then, the machine will only use IPv4s from its own subnet, ignoring traffic from other VLANs.
Thus, if the Pro Max Switch has more problems in that area, it could be an explanation. I told you back then:
QuoteI am at a loss why the clients stop renewing their privacy IPs. It cannot be RS, since they are never generated, it cannot be RAs, so it must be something that keeps the clients from assuming their new IPs, like DAD.
I found this one specific problem only because I saw multiple IPv6 prefixes all over the place when I used 802.1x.
Bug confirmed on "Pro Max" switch:
pro-max-bug.png
profile.png
I temporarily added a port profile with 802.1x enabled. A linux client did not exhibit the behavior but a Windows client did.
I think what's common with this bug and the earlier issue I mentioned where I was seeing multiple VLANs on Windows is that the port has both untagged and tagged VLANs on it. Windows apparently doesn't know what to do in that situation, maybe(?)
Quote from: meyergru on February 12, 2025, 10:58:33 AMThen, probably, those are based on RealTek chipsets, as well.
Yes, I think so. I see an "rtk.ko" kernel module loaded on the switch and a path "/usr/share/librtk..."
Quote from: meyergru on February 12, 2025, 03:03:45 PMThe SLAAC problems you are referring to are mentioned in this thread (https://forum.opnsense.org/index.php?topic=44435), which I already replied to. I have no problems with those.
Out of curiosity, do you keep your computer running 24/7 or do you shutdown and cause ethernet links to re-establish? Mine run constantly and I'm expecting the OS to take care of it.
QuoteAs I said, if you see multiple VLANs at once, IPv6 will definitely show problems, because then, multiple prefixes will get announced via RA.
I only had this issue of multiple VLANs in the beginning and had solved it long before I posted the thread about the temporary address generation issue. I don't think they're linked, unless something is still happening that is not visible.
Another variable is you are using an Enterprise switch with different internals than the Pro Max.
I had an Enterprise series switch, now have a Pro series. The problem only exists on my Pro.
For me, the visible VLANs all seem to be untagged with 802.1x. At least the effect is not limited to Windows. When I use 802.1x for a Fritzbox (Linux-based), I immediately see it picking up different DNS servers, all from different VLANs, see attached image.
Also, my Windows network adapter allows setting a VLAN tag. And if I leave that untagged and set the port to "trunk", I only get the untagged VLAN assignments. I also saw packets from all VLANs as untagged in a wireguard dump.
You can easily check: When you set your Windows port to trunk, do you see all assignments or only the untagged VLAN?
BTW: No matter if the other VLANs are tagged or untagged, it is still a security-relevant bug that you now confirmed to exist on the Pro Max series as well. I will update the info for Ubiquiti.
I'm unable to find an option to set a VLAN tag on my Windows network adapter, however I saw all of the VLAN assignments in my test. The port (as in my screenshot) was a trunk, with VLAN 30 as default/untagged and 'Allow all' for the others which makes them tagged in UniFi.