LACP on 10Gbit Ethernet-Card is only working with promiscuous mode enabled

[dm]

Hi,

I have a HP Proliant DL 380 Gen9 with 4x Broadcom NetXtreme BCM5719 Gigabit Ethernet (integrated NICs) and 4x Broadcom NetXtreme II BCM57810 10 Gigabit Ethernet (2x Dualport PCIe Cards installed). My switch is HPE FlexNetwork 5130 (JG941A).

I can't get LACP working with the 10Gbit Cards without enabling promiscuous mode in FreeBSD over ifconfig lagg0 promisc. With the 1Gbit Cards it's no Problem. There I do not have to enable promiscuous mode.

I tried two ports of the same 10 Gbit card in one LAGG and one port of each card. In both cases it doesent work. If only one ethernet cable is connectet it doesen't work ether.

Has anyone an idea how to fix this? I am also glad to hear about tips to troubleshoot this issue.

Thanks Darius

[mimugmail]

What happens without promic mode? No link? No bonding? Sounds weird.
On ix and ixl drivers it works without any workaround.

[dm]

Thanks for your reply.

Without promisc mode there is no bonding, but i have a link. Here are two outputs of my switch. The first without and the seccond with promisc mode enabled.

<switch>display link-aggregation verbose
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected, 
             I -- Individual, * -- Management port
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1
Aggregation Mode: Dynamic
Loadsharing Type: Shar
Management VLAN : None
System ID: 0x8000, 4cae-a365-4cdd
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  XGE1/0/51        U       32768    1         {ACD}
  XGE1/0/52        U       32768    1         {ACD}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag   
--------------------------------------------------------------------------------
  XGE1/0/51        1       32768    434       0x8000, 2c76-8a5c-7350 {ACG}
  XGE1/0/52        2       32768    434       0x8000, 2c76-8a5c-7350 {ACG}



<switch>display link-aggregation verbose
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected, 
             I -- Individual, * -- Management port
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1
Aggregation Mode: Dynamic
Loadsharing Type: Shar
Management VLAN : None
System ID: 0x8000, 4cae-a365-4cdd
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  XGE1/0/51        S       32768    1         {ACDEF}
  XGE1/0/52        S       32768    1         {ACDEF}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag   
--------------------------------------------------------------------------------
  XGE1/0/51        1       32768    434       0x8000, 2c76-8a5c-7350 {ACDEF}
  XGE1/0/52        2       32768    434       0x8000, 2c76-8a5c-7350 {ACDEF}


After disabling promisc mode I get this messages on the opnsense:

bxe0: Interface stopped DISTRIBUTING, possible flapping
bxe1: Interface stopped DISTRIBUTING, possible flapping

[Patrick M. Hausen]

The bnxt(4) driver in FreeBSD 11 has known bugs. VLANs don't work without explicit ifconfig promisc, either.
That should be fixed in FreeBSD 12 so waiting for the update to OPNsense 20.7 should be sufficient.

[dm]

Ok, thank you. We will wait for 20.7. and test it again.

[Patrick M. Hausen]

@dm Have you tried the new release already? How did it work out?

[dm]

I tried the new release and it worked perfect. Thanks again for your help.