Unifi "Pro" switches cannot handle layer 4 LACP distribution (and what follows)

[meyergru]

This is about a brand-new Unifi USW-Pro-HD-24-POE switch. I though I write this here to save others the time to find out themselves:

I recently bought one of those switches and wanted to use the LAGG feature with my OpnSense box with 4 I226V NICs.

My previous setup was to use two of those interfaces, one carrying my main (V)LAN and the other to carry all other VLANs, e.g. IoT. With that, I could have cross-VLAN traffic at 2.5 Gbps in both directions.

So I tried setting up a LAGG with LACP layer 4, in order to have TCP ports come into the mix for hashing and distributing different TCP streams over two physical links. I tested with "iperf3 -c x.x.x.x -P4" between two Linux VM hosts on different VLANs to have multiple source ports and to my great disbelief, got only 2.5 Gbps. The same thing happened when I tried against OpnSense itself.
I also tried two different iperf3 instances from the same machine to no avail.

After some more trial and error, I found that the Unifi switch can only handle layer 2 & 3 hashing, which is especially disappointing when you know that they did better years ago. Only when I had two different clients do independent runs, could I have > 4 Gbps in sum - and even this took a few tries before I found a counterpart machine with a matching MAC.

I then found this tidbit in the CLI of the switch:

Code Select Expand

BusyBox v1.25.1 () built-in shell (ash)


  ___ ___      .__________.__
 |   |   |____ |__\_  ____/__|
 |   |   /    \|  ||  __) |  |   (c) 2010-2024
 |   |  |   |  \  ||  \   |  |   Ubiquiti Inc.
 |______|___|  /__||__/   |__|
            |_/                  https://www.ui.com

      Welcome to UniFi USW-Pro-HD-24-PoE!

********************************* NOTICE **********************************
* By logging in to, accessing, or using any Ubiquiti product, you are     *
* signifying that you have read our Terms of Service (ToS) and End User   *
* License Agreement (EULA), understand their terms, and agree to be       *
* fully bound to them. The use of SSH (Secure Shell) can potentially      *
* harm Ubiquiti devices and result in lost access to them and their data. *
* By proceeding, you acknowledge that the use of SSH to modify device(s)  *
* outside of their normal operational scope, or in any manner             *
* inconsistent with the ToS or EULA, will permanently and irrevocably     *
* void any applicable warranty.                                           *
***************************************************************************

hagen-US3.7.1.33# cli
hagen# configure
hagen(config)# lag load-balance
Incomplete command
hagen(config)# lag load-balance ?
  src-dst-mac     LAG load balancing is based on source and destination MAC addr
                  ess.
  src-dst-mac-ip  LAG load balancing is based on source and destination of MAC a
                  nd IP addresses.
hagen(config)#

This shows that only layer 2 & 3 are being supported, as my tests confirmed.

Actually, for my scenario, LAGGs are worse without layer 4 hashing than splitting the VLANs across the interfaces, as I would have to be lucky (chances are 50:50) to have two arbitrary devices be put on different links via layer 2 or 3 hashing, even when I use multiple TCP streams with different ports.

I also posted this to the Unifi forum.

Oh, BTW: Today, I looked at my USW-Enterprise-24-PoE. It has a completely different CLI structure and obviously, this one does in fact support LACP layer 4:

Code Select Expand

BusyBox v1.25.1 () built-in shell (ash)


  ___ ___      .__________.__
 |   |   |____ |__\_  ____/__|
 |   |   /    \|  ||  __) |  |   (c) 2010-2024
 |   |  |   |  \  ||  \   |  |   Ubiquiti Inc.
 |______|___|  /__||__/   |__|
            |_/                  https://www.ui.com

      Welcome to UniFi USW-Enterprise-24-PoE!

********************************* NOTICE **********************************
* By logging in to, accessing, or using any Ubiquiti product, you are     *
* signifying that you have read our Terms of Service (ToS) and End User   *
* License Agreement (EULA), understand their terms, and agree to be       *
* fully bound to them. The use of SSH (Secure Shell) can potentially      *
* harm Ubiquiti devices and result in lost access to them and their data. *
* By proceeding, you acknowledge that the use of SSH to modify device(s)  *
* outside of their normal operational scope, or in any manner             *
* inconsistent with the ToS or EULA, will permanently and irrevocably     *
* void any applicable warranty.                                           *
***************************************************************************

edgar-US.7.1.26# cli

Entering character mode
Escape character is '^]'.

Warning!
The changes may break controller settings and only be effective until reboot.

(UBNT) >enable

(UBNT) #configure

(UBNT) (Config)#port-channel load-balance ?

1                        Src MAC, VLAN, EType, incoming port
2                        Dest MAC, VLAN, EType, incoming port
3                        Src/Dest MAC, VLAN, EType, incoming port
4                        Src IP and Src TCP/UDP Port fields
5                        Dest IP and Dest TCP/UDP Port fields
6                        Src/Dest IP and TCP/UDP Port fields

(UBNT) (Config)#port-channel load-balance

So, it seems model-dependend, probably based on product lines (i.e. Pro vs. Enterprise).