What is LAGG's impact on LAN performance - load balance protocol?

[Shoresy]

Being new to OPNsense...I had two 2.5Gbe ports unused on my mini PC and decided to allocate them to a LAGG interface within OPNsense, then set my assigned LAN interface to the LAGG group. There are now 3 interfaces assigned to the LAGG group in total. Unfortunately my managed switch is rather dumb and doesn't support LACP (it does support simple static lag though), so I opted for the load balancing protocol for the LAGG group in OPNsense and ran 2 extra patch cables to the managed switch, setting those 3 ports to static LAGG on the switch.

Everything is working fine and I can see packets being evenly distributed over the 3 switch ports (within the switch's user interface) set to LAGG load balance mode. Eventually I'll replace that managed switch with one that supports 802.3ad, but was hoping someone more knowledgeable could chime in and comment on whether my existing load balancing setup (described above) is adding any benefit or not for my LAN. Just figured it was better to use the open ports on the OPNsense box rather than leave them sitting inactive. My home network is simple, no VLANs. Would round robin or another protocol be a better option?

[Koldnitz]

I ran a lagg (lacp) for a little while.

There were no real gains to be had, it added complexity and I noticed interface errors (they were not detrimental to performance).

There is often a misunderstanding on how lagg (lacp works).

It just makes the pipe bigger.  Your stream (I am calling a stream the data connection between a single computer and the switch / router ) is not split between the 2 ports if you are fully saturating one of the 2.5 GB ports.

If your network (more than one computer / device) has the bandwidth to fully saturate one 2.5G port then you might see a benefit, but if you have all 2.5 GB ports I'm assuming your modem connection is 2.5 as well so then the bottleneck will be the modem.

I do run a lagg (lacp) on my NAS and it "should" allow me to use both ports if multiple devices are accessing it at once (who knows if it actually benefits me; the NAS has multiple ports).

The consensus on these forums seems to be that it adds complexity for no real gains.

However if you are actually fully saturating the 2.5G port between your switch and your router and you are doing this with more than one device (multiple streams - this may be incorrect terminology btw) you might gain a benefit from it.

From what I understand there are overheads, so lagging two 1GBE ports does not give you 2G bandwidth.

Cheers,

[Shoresy]

Thanks @Koldnitz for the detailed explanation. Agree that I'm not seeing any real performance gains (so far) using lagg, and the devices on my network are not saturating those 2.5Gbe ports. Logically you'd think widening the pipe would at least add some efficiency gains by spreading the load out to multiple ports, but I would imagine the processing and complexity overhead likely negates any real benefit from doing so. I'll likely end up undoing the lagg config...wanted to run it for a few days just to see if there was any real impact. So far, nothing noticeable, though at least there haven't been any negative impacts that I've observed. Appreciate the reply!

[Patrick M. Hausen]

There is no noticeable processing ovehead.
LACP is the only reasonable way to do it, because everything else, especially round-robin might lead to TCP packets arriving out of order. That is a bad thing.
Also no switch I know supports round-robin in the other direction.
You will get at maximum the peak bandwidth pf a single interface for a single stream. This is by design.
If you have a hundred internal users/PCs, then on average 50 will use each link in a two link lagg.
Lagg is not overly complicated. It is used in millions of enterprise environments daily.
Think of a firewall in front of a server farm with hundreds of thousands of requests coming in.

To debunk some of the nonsense in this thread.

Home networks are not the primary use of firewalls. Do you need lagg at home? No. Do you want it to learn about networks? Go ahead, it does not hurt in any way.

Your switch must support LACP, though. LACP is the only way to fly.

[Koldnitz]

I fully defer to what pmhausen says on this.

A lot of my limited understanding on lagg (lacp) is due to reading his and some of the other professional types posts on the subject.

The overheads I spoke of were minimal I believe around or slightly less than 5 percent (I think I read this on the pfsense forums while researching the subject).

Cheers,

[Vilhonator]

LAGG is important with NAS systems and servers.

As mentioned earlier, it gives you benefits when connections fully saturate the ports.

Think it as this way, when you are hosting NAS with 2 10Gb ports, how likely will you have even 30 computers with 10Gb ports downloading and uploading stuff to it non-stop? Hardly any of your devices do that more than once or twice for few minutes a day if even that, also hard drives, SSDs or read/write storage would have to be fast enough to exceed the port speed as well (which is possible with using M.2 NVME SSD storage).

LAGG in home use could be useful, when you have multiple computers and TVs with streaming service like netflix capabilities, because then full saturation is more likely to happen, assuming your internet speed is fast enough to offer optimal bandwidth to every device at same time and amount of devices exceeds the network port speeds they are connected to (just 300Mb/s connection can provide theoratically optimal connection for full HD up to 12 devices, so you would have to have quite a few devices at home streaming netflix and other internet stuff at the same time to even require LAGG, though adding NAS with 10Gb connection can change that).

So no, most home users don't see any benefit from using LAGG, the benefit is notable, when you for example work at school or some company, and have to setup NAS and other network services for students and employees to use

This is at least how I understand it to be, and I could be wrong

[Shoresy]

Great responses, very informative. It's a real bummer my managed switch doesn't support LACP, would've liked to see it in action even though for home-use purposes I don't push the envelope remotely hard enough to reap the benefits. I'm going to drop the lagg config since I can't use the lacp protocol anyway. It was great to learn more thoroughly about when and why it should be used, so thanks!

Edit: I do have a 24-port LACP-capable PoE+ switch on the way, so I'll revisit lagg once I swap the other one out.

[Vilhonator]

You're welcome

If you want to test out LAGG and buying new switch for it is too expensive for your budget, you can look for used Cisco Switches (catalyst 2960 series for example is great, though might cost a pretty penny even as used, the new 1000 series is exactly the same, but compared to 2960 series, lot cheaper).

Also if you have a computer with enough SATA ports lying around, you can install either Linux, windows or TrueNAS server to it (You can download evaluation version of windows server for free, which is great option unless you look for something permanent), and test LAGG that way (with 4 regular SATA III SSDs set to RAID 5 or 6, you will be exceeding speeds of 1Gb ports, not sure about 10Gb, but 1Gb for sure) and test it out.

Only way how home users are able to fully saturate even 1Gb ports, is by uploading and downloading by far over 120MB worth of files from their NAS (1Gb = 1 Gigabit = ~ 120 MB. Capital letter always matter when used to describe measurements) as long as server has storage which can read and write stuff faster than is downloaded or written.

[Shoresy]

I dropped in the new managed switch this afternoon, finally configured the LAGG group using the LACP protocol. Things seem similar, performance-wise, to load balance mode. I hadn't experienced out-of-order packets when load balancing between OPNsense and the previous managed switch, but I'm sure load balancing is less robust compared to LACP. So far connectivity seems to be working well between the router and switch...I can run two 1Gbps downloads simultaneously over both 1Gb WANs downstream with full 1Gbps throughput to the LAN, whereas previously two large 1Gbps downloads flowing in from both WANs over one link would've undoubtedly been halved based on limitations of a single 1Gbps full duplex port (on the switch). The load balancing seems to work intelligently enough to fork those 1Gbps downloads across both WANs, then over to the switch using dedicated links. It's overkill for sure as that scenario rarely occurs on my network.

[Shoresy]

Even though this scenario doesn't happen often, I wanted to test 2 independent speedtests running on 2 separate devices on my LAN (to the same speedtest server), forced out over a different WAN on each device, to find out if I could achieve > 1Gbps thanks to the LACP LAGG between the OPNsense router/switch. The 2 shades of red are the different WANs, downstream of nearly 1Gbps each.

Even though this isn't the best screenshot, it does demonstrate that LACP LAGG pushed the throughput out over the LAN interface to ~1.6Gbps (the tag on the screenshot was just before the crest of about 1.8Gbps), thanks to the LAGG between the router/1Gb managed switch.