LAGG Setup (in OPNsense vs. Switch)

[fakebizprez]

I have a network topology question regarding LAGG configuration. My setup is:

OPNsense firewall/router (handling firewall, DHCP, DNS, Wireguard)
Downstream: Cisco Catalyst 3850 and Cisco Nexus 9000 C92160YC-X switches
Multiple servers connected to the Cisco switches

I'm planning to implement LAGG for my downstream servers for redundancy and increased bandwidth. My question is:
When setting up LAGG for servers that are connected to the Cisco switches (not directly to OPNsense), where should I configure the LAGG interfaces? Should this be configured on:

• The OPNsense server, or
• The corresponding Cisco switch that the server is directly connected to

I understand LAGG typically requires configuration on both ends of a connection, but I'm unclear about the role OPNsense plays when the servers are connected through downstream switches rather than directly to OPNsense.

Any guidance on the proper approach for this topology would be greatly appreciated.

Thanks in advance!

[EricPerl]

I've never experimented with LAGG but I'd be surprised if anything but both ends of the aggregation were aware.

[Patrick M. Hausen]

LACP is a link layer protocol that is "spoken" on the direct links between the connected devices.

So if you want to connect OPNsense with a lagg interface to your switch you must configure this on OPNsense and the switch, respectively for the ports that are supposed to form the lagg/port-channel (the latter being Cisco-ese for lagg).

If you want to connect a server with a lagg you need to configure this on the server and the switch.

Neither server nor OPNsense care or can even detect if any other device is connected via a lagg. This stays between the switch and the device in question. Ideally all essential systems, servers *and* OPNsense are connected with a lagg to *two* switches that can do multi chassis LACP or "stacking" as Cisco sometimes calls it.

HTH,
Patrick

[meyergru]

And do not be too surprised if this does work, but does nothing that you potentially expect.

LACP knows several levels of traffic distribution, based on combinations of source and destination IPs and ports. The actual implementation is at the discretion of the manufacturer. Some devices do not support the most desireable distribution types. That can result in traffic not being distributed over multiple links, but staying on the same one. This is especially true for single TCP connections between two partners at both ends of the LAGG: since their IPs and source/destination ports are fixed within one session, the bandwidth will effectively stay at the speed of one link only - and this is true, regardless of good or bad implementation.

So, when you do an iperf measurement with only one stream over a LAGG, you will have no speed increase compared with a single link.

The rule of thumb is: LAGGs help only to "statistically" distribute traffic for many clients, not for any specific 1:1 connection. So, essentially, it does not help much at all for home users, more so in a business/enterprise context.