[SOLVED] Performance tuning a 2x10Gbps LACP LAG

[AWBbox]

Hi everyone,

I've been a pfSense user for many years and fancied giving OPNsense a try for comparison. I'm having some throughput issues with LACP. As the attached diagram shows, I have a 4x10Gbps LACP LAG between my ESXi hypervisor and the switch, and a 2x10Gbps LACP LAG between the switch and the OPNsense firewall.

I have multiple VLANs and subnets in this topology, with the firewall being the gateway on which they all reside. iperf3 results between hosts on the hypervisor on the same subnet are ridiculously fast because the traffic doesn't leave the physical network interface, only staying within the DSwitch. iperf tests between subnets however will inevitably have to be routed via the firewall and this is where I am having problems.

Throughput is capping out at around 16Gbps with OPNsense CPU usage at only 30% and I would like to see it nearer 19Gbps. I have read guides such as https://calomel.org/freebsd_network_tuning.html around performance tuning, disabled hardware CRC, TSO, LRO, and applied many tunable variables which do not appear to have made any impact:

Code: [Select]

hw.ibrs_disable="1"
if_ixl_updated_load="1"
kern.ipc.maxsockbuf="16777216"
net.inet.ip.maxfragpackets="0"
net.inet.ip.maxfragsperpacket="0"
net.inet.rss.enabled="1"
net.inet.tcp.abc_l_var="44"
net.inet.tcp.cc.abe="1"
net.inet.tcp.initcwnd_segments="44"
net.inet.tcp.isn_reseed_interval="4500"
net.inet.tcp.minmss="536"
net.inet.tcp.mssdflt="1460"
net.inet.tcp.recvbuf_max="4194304"
net.inet.tcp.recvspace="65536"
net.inet.tcp.rfc6675_pipe="1"
net.inet.tcp.sendbuf_inc="65536"
net.inet.tcp.sendbuf_max="4194304"
net.inet.tcp.sendspace="65536"
net.inet.tcp.soreceive_stream="1"
net.inet.tcp.syncache.rexmtlimit="0"
net.inet.tcp.syncookies="0"
net.inet.tcp.tso="0"
net.inet6.ip6.maxfragpackets="0"
net.inet6.ip6.maxfrags="0"
net.isr.bindthreads="1"
net.isr.defaultqlimit="8192"
net.isr.dispatch="deferred"
net.isr.maxthreads="-1"
net.link.lagg.default_use_flowid="1"
net.pf.source_nodes_hashsize="1048576"

I am using an Intel XL710 card (Supermicro AOC-STG-i4S) in OPNsense, using the latest firmware and latest Intel drivers https://www.freshports.org/net/intel-ixl-kmod/ as opposed to the ones that come with the OS out of the box. Both NICs in the LAG appear as follows:

Code: [Select]

ixl2: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.40> mem 0x60e0800000-0x60e0ffffff,0x60e2808000-0x60e280ffff at device 0.2 on pci1
ixl2: using 1024 tx descriptors and 1024 rx descriptors
ixl2: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d2ab oem 1.268.0
ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
ixl2: Using MSI-X interrupts with 9 vectors
ixl2: Allocating 8 queues for PF LAN VSI; 8 queues active
ixl2: Ethernet address: ac:1f:6b:8d:08:ae
ixl2: PCI Express Bus: Speed 8.0GT/s Width x8
ixl2: SR-IOV ready
ixl2: The device is not iWARP enabled
ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
ixl2: link state changed to UP
ixl2: TSO4 requires txcsum, disabling both...
ixl2: TSO6 requires txcsum6, disabling both...
ixl3: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.12.40> mem 0x60e0000000-0x60e07fffff,0x60e2800000-0x60e2807fff at device 0.3 on pci1
ixl3: using 1024 tx descriptors and 1024 rx descriptors
ixl3: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d2ab oem 1.268.0
ixl3: PF-ID[3]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
ixl3: Using MSI-X interrupts with 9 vectors
ixl3: Allocating 8 queues for PF LAN VSI; 8 queues active
ixl3: Ethernet address: ac:1f:6b:8d:08:af
ixl3: PCI Express Bus: Speed 8.0GT/s Width x8
ixl3: SR-IOV ready
ixl3: The device is not iWARP enabled
ixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
ixl3: link state changed to UP
ixl3: TSO4 requires txcsum, disabling both...
ixl3: TSO6 requires txcsum6, disabling both...

[admin@lonrtr01 ~]$ ifconfig -vvvv lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=600000a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VXLAN_HWCSUM,VXLAN_HWTSO>
        ether ac:1f:6b:8d:08:ae
        laggproto lacp lagghash l4
        lagg options:
                flags=15<USE_FLOWID,USE_NUMA,LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 2
                flapping: 0
        lag id: [(8000,AC-1F-6B-8D-08-AE,0152,0000,0000),
                 (8000,F0-9F-C2-0C-85-F8,001A,0000,0000)]
        laggport: ixl2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,AC-1F-6B-8D-08-AE,0152,8000,0003),
                 (8000,F0-9F-C2-0C-85-F8,001A,0080,0001)]
        laggport: ixl3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,AC-1F-6B-8D-08-AE,0152,8000,0004),
                 (8000,F0-9F-C2-0C-85-F8,001A,0080,0002)]
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

I was hoping I might be able to get some input from the community as to what I can do to squeeze the last few Gbps out of this LAG!

[lilsense]

I recommend enabling Receive Side Scaling.
https://forum.opnsense.org/index.php?topic=24409.0

[AWBbox]

Thank you for your response, however Receive Side Scaling is already enabled as per net.inet.rss.enabled="1". This has been tested and verified on my part, OPNsense was only sending data on one of the two links prior to this, resulting in only 9.5Gbps transfer speeds.

[AWBbox]

I wanted to provide an update to this in case anyone else looks for this solution in future. A kind soul on Reddit provided the answer in the form of jumbo frames!

Increased MTU size to 9000 across my topology and the attached screenshot results. This traffic is being routed and firewalled between the two subnets shown, making me quite happy