Static LAGG working only partially : not with VLAN or CARP

[toxic]

Hello, I did a nice setup and I was expecting that my 2 windows clients could get full gigabit speeds using SMBv3 to my NAS, and somehow they still share the bandwidth, I don't get why...

TLDR: Loadbalancing on a 2 NIC Group without VLAN from router to NAS through the switch is working fine. But Router has another LAG group of 3 NIC to several tagged VLAN on the switch, but on the same VLAN, 2 clients on 2 different ports share a single GB/s to the router despite the router having LAGG...


My setup is :

• TL-SG1024DE 24ports Gigabit Switch (no LACP...)
• synology NAS with 2 plugs on static LAGG group on VLAN 11 (untagged&PVID11 on the switch)
• OpnSense router with 5 NIC : 2 plugged in a static LAGG group on VLAN 11), 3 in another static LAGG group with several VLANs on it, but let's focus on VLAN 30
• Client A Windows laptop plugged in on the switch in a port set to VLAN30 untagged and PVID 30
• Client B Ubuntu laptop plugged  in on the switch in a different port also set to VLAN30 untagged and PVID 30

Both client A and B get a proper IP from router R on the VLAN30, each individually is able to copy files using SMBv3 at 100-110 Mo/s when it's the only one do to so.
But when both client A and B are trying to download a file using SMBv3 at the same time, their combined speed never gets to 2BG/s, it stays at 1GB/s only.

2 strange things happens that tell me somehow I'm not so dumb and LAGG is not too badly configured :
- When I plug client B on a different port on the switch that is directly attached to VLAN11 like the NAS, I can indeed hit the NAS from client A and B at the same time using SMBv3 and total bandwith seen from the NAS is 2GB/s => so SMBv3 protocol is not the issue and is fast enough to show that I use LAG to get past 1GB/s is working properly
- Now stranger even : I started an iperf server on the NAS, and when I use iPerf as client on the opnSense router, using --reverse, I am able to get 1GB/s bandwith, and while it is running, I am able to use client A to start a SMBv3 download and also get 110Mo/s at the same time, so the nas realy sees at this time that it has 2BG/s of traffic outgoing ! So LAGG between router and NAS is indeed working and balancing the load allowing 2GB/S if more than 1 client.

Now, last leg that could be not working is the LAG from my router to my switch on the 3 port group to all different VLANS... And indeed : when I start iperf server on the router, use client B as iperf client --reverse, I do see 1GB/s, but I see it drop when client A uses the router also, for example to download the file from the NAS (client A is windows I don't have iperf on it...)

So the issue seems to be that my switch is not able to loadbalance to the 3 port LAG group I setup for connectivity to the router on all my VLANS.

Now, I do use CARP so my router which is the default gateway is beeing contacted throuth the CARP IP and not it's real IP, but I doubt that could be a factor for this LAGG to not work.

I setup the 3ports LAGG pretty much the same way as the 2ports one, only adding VLAN :
- lagg0 type LOADBALANCE on igb2, igb3 and igb4 (somehow the opnSense GUI in the OtherTypes->LAGG show the same MAC address for all these... but is does the same for the 2 port LAG that is working)
- created a butload of VLAN on lagg0
- assign "vlan 30 on lagg0" to lan or opt1, static IPv4

Did I missed something ?

Thank you in advance for your help and thanks a lot anyway for your reading !

If you read this far, you probably want some details :
ifconfig -a : (I left out several lagg0_vlanXX that looked the same anyway)

Code Select Expand


igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=800028<VLAN_MTU,JUMBO_MTU>
ether 00:40:d7:e0:09:c2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=800028<VLAN_MTU,JUMBO_MTU>
ether 00:40:d7:e0:09:c2
hwaddr 00:40:d7:e0:09:c3
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
ether 10:10:10:10:10:01
inet 10.0.10.2 netmask 0xffffff00 broadcast 10.0.10.255
inet 10.0.10.1 netmask 0xffffffff broadcast 10.0.10.1 vhid 10
inet6 fe80::1210:10ff:fe10:1001%vtnet0 prefixlen 64 scopeid 0x3
groups: GR_LAN_Servers
carp: MASTER vhid 10 advbase 1 advskew 0
media: Ethernet 10Gbase-T <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=800028<VLAN_MTU,JUMBO_MTU>
ether 00:40:d7:e0:09:c4
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=800028<VLAN_MTU,JUMBO_MTU>
ether 00:40:d7:e0:09:c4
hwaddr 00:40:d7:e0:09:c5
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=800028<VLAN_MTU,JUMBO_MTU>
ether 00:40:d7:e0:09:c4
hwaddr 00:40:d7:e0:09:c6
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
groups: enc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33160
groups: pflog
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 9000
pfsync: syncdev: lagg0_vlan9 syncpeer: 10.0.9.3 maxupd: 128 defer: off
groups: pfsync
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=800028<VLAN_MTU,JUMBO_MTU>
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0 prefixlen 64 scopeid 0xb
laggproto loadbalance lagghash l2,l3,l4
laggport: igb2 flags=4<ACTIVE>
laggport: igb3 flags=4<ACTIVE>
laggport: igb4 flags=4<ACTIVE>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=800028<VLAN_MTU,JUMBO_MTU>
ether 00:40:d7:e0:09:c2
inet6 fe80::240:d7ff:fee0:9c2%lagg1 prefixlen 64 scopeid 0xc
inet 10.0.11.2 netmask 0xffffff00 broadcast 10.0.11.255
inet 10.0.11.1 netmask 0xffffffff broadcast 10.0.11.1 vhid 11
laggproto loadbalance lagghash l2,l3,l4
laggport: igb0 flags=4<ACTIVE>
laggport: igb1 flags=4<ACTIVE>
groups: lagg GR_LAN_Servers
carp: MASTER vhid 11 advbase 1 advskew 0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan30 prefixlen 64 scopeid 0xd
inet 10.0.30.2 netmask 0xffffff00 broadcast 10.0.30.255
inet 10.0.30.1 netmask 0xffffffff broadcast 10.0.30.1 vhid 30
groups: vlan GR_LAN_Clients
carp: MASTER vhid 30 advbase 1 advskew 0
vlan: 30 vlanpcp: 2 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan3 prefixlen 64 scopeid 0xe
inet6 2a01:e0a:336:6ea0:240:d7ff:fee0:9c4 prefixlen 64 autoconf
inet 192.168.1.3 netmask 0xffffff00 broadcast 192.168.1.255
inet 192.168.1.2 netmask 0xffffffff broadcast 192.168.1.2 vhid 3
groups: vlan GR_WAN
carp: MASTER vhid 3 advbase 1 advskew 0
vlan: 3 vlanpcp: 2 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lagg0_vlan9: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan9 prefixlen 64 scopeid 0xf
inet 10.0.9.2 netmask 0xffffff00 broadcast 10.0.9.255
inet 10.0.9.1 netmask 0xffffffff broadcast 10.0.9.1 vhid 9
groups: vlan GR_LAN_Servers
carp: MASTER vhid 9 advbase 1 advskew 0
vlan: 9 vlanpcp: 7 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan11 prefixlen 64 scopeid 0x10
groups: vlan
vlan: 11 vlanpcp: 3 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan22: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan22 prefixlen 64 scopeid 0x11
inet 10.0.22.2 netmask 0xffffff00 broadcast 10.0.22.255
inet 10.0.22.1 netmask 0xffffffff broadcast 10.0.22.1 vhid 22
groups: vlan GR_LAN_NoAccess
carp: MASTER vhid 22 advbase 1 advskew 0
vlan: 22 vlanpcp: 1 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan40: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan40 prefixlen 64 scopeid 0x12
inet 10.0.40.2 netmask 0xffffff00 broadcast 10.0.40.255
groups: vlan GR_LAN_Clients
vlan: 40 vlanpcp: 2 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan50 prefixlen 64 scopeid 0x13
inet 10.0.50.2 netmask 0xffffff00 broadcast 10.0.50.255
groups: vlan GR_LAN_Clients
vlan: 50 vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan60: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan60 prefixlen 64 scopeid 0x14
inet 10.0.60.2 netmask 0xffffff00 broadcast 10.0.60.255
groups: vlan GR_LAN_Clients
vlan: 60 vlanpcp: 1 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan1 prefixlen 64 scopeid 0x15
inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255
groups: vlan GR_LAN_NoAccess
vlan: 1 vlanpcp: 1 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan8: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan8 prefixlen 64 scopeid 0x16
inet 10.0.8.2 netmask 0xffffff00 broadcast 10.0.8.255
groups: vlan GR_LAN_Servers
vlan: 8 vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan99: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
ether 00:40:d7:e0:09:c4
inet6 fe80::240:d7ff:fee0:9c4%lagg0_vlan99 prefixlen 64 scopeid 0x17
inet 10.0.99.2 netmask 0xffffff00 broadcast 10.0.99.255
groups: vlan GR_LAN_NoAccess
vlan: 99 vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


[EDIT] :
I noticed that I don't have any VLAN untagged on this LAG group in the switch, and as such I also had not assigned the lagg0 interface to any optXX. So I thought maybe lagg0 is not "properly enabled", I added an OPT11 to assign lagg0, enabled it but left both IPv4 and IPv6 to "None", and I also did that for a vlan I declared on lagg 0 but never got to use.... So I'm posting the full ifconfig -a just in case...