Can't make a LAGG interface work properly

[ThomasRicou]

Hi,
I'm testing OPNSense to replace my actual second level firewalls (not the one connected to the internet but between my public network and intern networks).

I configured a LAGG with 2 interfaces in LACP, on my cisco 3750 switch I configured the 2 matching ports in a channel group :

- bxe3 and bxe2 are the two physical interfaces plugged in gi2/0/46 and gi2/0/47 (same order) which are aggregated in port-channel 12 :

Code: [Select]

interface GigabitEthernet2/0/46
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 12 mode on
end

interface GigabitEthernet2/0/47
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 12 mode on
end

interface Port-channel12
 switchport trunk encapsulation dot1q
 switchport mode trunk
end

- LAGG lagg0 with members bxe2 and bxe3 and LACP protocol


I configured a static IPv4 address on my LAN (lagg0) interface but even if my switch is telling me that the ports are bundled, it does not work. Actually I can't see any packet between the FW and the switch.

Any idea ?

For now, I managed to lock myself out, I'm gonna start over tomorrow :-(

[mimugmail]

Change to Mode access when you don't use trunking at the Firewall

[Alphabet Soup]

mimugmail has probably solved it... as you don't mention any VLANs, why are you trunk'ing?  Or, until you permit any VLANs, what are you trunking?

That aside, I have two differences in my 3750E / 3850 GigabitEthernet interface configs when aggregating to FreeBSD servers:

Code: [Select]

channel-protocol lacp
channel-group 12 mode active
Maybe the defaults for your IOS version don't require this anymore, but unless I force LACP and force ACTIVE, it doesn't work for me.

[ThomasRicou]

Sorry, I've forgotten to precise that I was using Vlans...
I'll try without. (I'm would think I did it already but I ran though so many tests I can't remember...)
I'll let you know...

[ThomasRicou]

Hi,

It doesn't work :
interface GigabitEthernet2/0/46
 switchport access vlan 1001
 switchport mode access
 channel-protocol lacp
 channel-group 12 mode active
end

interface GigabitEthernet2/0/47
 switchport access vlan 1001
 switchport mode access
 channel-protocol lacp
 channel-group 12 mode active
end

interface Port-channel12
 switchport access vlan 1001
 switchport mode access
 macro description serverport
 spanning-tree portfast
end

        w - waiting to be aggregated
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------
12     Po12(SU)        LACP      Gi2/0/46(w) Gi2/0/47(w)

As you can see, the LACP aggregation is not completing.

Any idea on how to debug that ?

[mimugmail]

Dont use spanning tree portfast and turn in debug lacp on Catalyst

[Alphabet Soup]

Yeah, the "spanning-tree portfast" is OK if the node at the other end is a leaf (e.g. a single server), not a branch (e.g. a switch with multiple endpoints beyond).  Not knowing anything about your OPNsense configuration, it would be safer not to consider it a leaf, and remove the "spanning-tree portfast" for now.

One other trick on the 3750 side is after you have finished configuring the physical ports and virtual port-channel, to "shutdown" then "no shutdown" the physical ports to make IOS really notice your changes.

Something like:

Code: [Select]

configure terminal
interface range GigabitEthernet 2/0/46-47
shutdown
{ wait a few seconds }
no shutdown
exit
exit
Or reboot the whole switch.

[ThomasRicou]

Hi,
Thx for your replies.
I 've unset the STP portfast, shut/no shut the ports and even unplug/plug the ports but nothing changed. In the debug mode, I'm not an expert but the logs indicate that the ports are alternatively ready/not ready for entering the LACP LAGG.
It starts with :

Code: [Select]

1342182: Aug 24 10:28:23.047: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/46, changed state to down
1342183: Aug 24 10:28:23.081: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/47, changed state to down
1342184: Aug 24 10:28:25.136: FEC: lacp_switch_add_port_to_associated_list_internal: Gi2/0/46 added to list for Po12
1342185: Aug 24 10:28:25.144: FEC: lacp_switch_add_port_to_associated_list_internal: Gi2/0/47 added to list for Po12
1342186: Aug 24 10:28:26.839: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342187: Aug 24 10:28:26.839: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Gi2/0/46 associated to Po12
1342188: Aug 24 10:28:26.839: FEC: lacp_switch_is_port_in_associate_list: port Gi2/0/46 is present in the associate list
1342189: Aug 24 10:28:26.839: FEC: lacp_switch_get_next_associated_port_from_agg_id: found port Gi2/0/47 next to Gi2/0/46 and associated to Po12
1342190: Aug 24 10:28:26.839: FEC: lacp_switch_is_port_in_associate_list: port Gi2/0/47 is present in the associate list
1342191: Aug 24 10:28:26.839: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Gi2/0/47 in aggregator Po12
1342192: Aug 24 10:28:26.839: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342193: Aug 24 10:28:26.839: FEC: lacp_switch_check_hw_sw_constraints_internal: port Gi2/0/46 can be bundled in the aggregator Po12, new afb->nports [0]
1342194: Aug 24 10:28:27.023: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/46, changed state to up
1342195: Aug 24 10:28:27.023: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/47, changed state to up
1342196: Aug 24 10:28:27.031: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342197: Aug 24 10:28:27.031: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Gi2/0/46 associated to Po12
1342198: Aug 24 10:28:27.031: FEC: lacp_switch_is_port_in_associate_list: port Gi2/0/46 is present in the associate list
1342199: Aug 24 10:28:27.031: FEC: lacp_switch_get_next_associated_port_from_agg_id: found port Gi2/0/47 next to Gi2/0/46 and associated to Po12
1342200: Aug 24 10:28:27.031: FEC: lacp_switch_is_port_in_associate_list: port Gi2/0/47 is present in the associate list
1342201: Aug 24 10:28:27.031: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Gi2/0/47 in aggregator Po12
1342202: Aug 24 10:28:27.031: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342203: Aug 24 10:28:27.031: FEC: lacp_switch_check_hw_sw_constraints_internal: port Gi2/0/47 can be bundled in the aggregator Po12, new afb->nports [0]
1342204: Aug 24 10:28:27.241: FEC: lacp_switch_remove_port_from_associated_list_internal: Gi2/0/46 deleted from the associated list for Po12

I can't get the exact following logs as it's too fast but it repeats with some kind of :

Code: [Select]

1342546: Aug 24 10:31:11.849: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Gi2/0/46 in aggregator Po12
1342547: Aug 24 10:31:11.849: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342548: Aug 24 10:31:11.849: FEC: lacp_switch_check_hw_sw_constraints_internal: port Gi2/0/46 can be bundled in the aggregator Po12, new afb->nports [1]
1342549: Aug 24 10:31:13.711: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342550: Aug 24 10:31:13.711: FEC: add port (Gi2/0/46) to agport (Po12)
1342551: Aug 24 10:31:13.711: FEC: pagp_switch_add_port_to_agport_list: afb->nports++ = 2 [Gi2/0/46]
1342552: Aug 24 10:31:13.711: FEC: lacp_switch_add_port_to_agport_internal: Gi2/0/46 added to aggregator Po12 list
1342553: Aug 24 10:31:13.711: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342554: Aug 24 10:31:13.711: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Gi2/0/47 associated to Po12
1342555: Aug 24 10:31:13.711: FEC: lacp_switch_is_port_in_associate_list: port Gi2/0/47 is present in the associate list
1342556: Aug 24 10:31:13.711: FEC: lacp_switch_get_next_associated_port_from_agg_id: found port Gi2/0/46 next to Gi2/0/47 and associated to Po12
1342557: Aug 24 10:31:13.711: FEC: lacp_switch_is_port_in_associate_list: port Gi2/0/46 is present in the associate list
1342558: Aug 24 10:31:13.711: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Gi2/0/46 in aggregator Po12
1342559: Aug 24 10:31:35.875: FEC: lacp_switch_display_oneline: found 1 aggregators
1342560: Aug 24 10:31:35.884: FEC: lacp_switch_display_oneline: found 2 ports
1342561: Aug 24 10:31:36.001: FEC: lacp_switch_delete_port_from_agport_internal: removing Gi2/0/47 from Po12
1342562: Aug 24 10:31:36.001: FEC: delete port (Gi2/0/47) from agport (Po12)
1342563: Aug 24 10:31:36.001: FEC: pagp_switch_delete_port_from_agport_list: afb->nports-- = 1 [Gi2/0/47]
1342564: Aug 24 10:31:36.001: FEC: lacp_switch_remove_port_from_associated_list_internal: Gi2/0/47 deleted from the associated list for Po12
1342565: Aug 24 10:31:36.001: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342566: Aug 24 10:31:36.009: FEC: pagp_switch_reset_load_index: reading load-index for port Po12
1342567: Aug 24 10:31:36.068: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342568: Aug 24 10:31:36.068: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Gi2/0/46 associated to Po12
1342569: Aug 24 10:31:36.068: FEC: lacp_switch_is_port_in_associate_list: port Gi2/0/46 is present in the associate list
1342570: Aug 24 10:31:36.068: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Gi2/0/46 in aggregator Po12
1342571: Aug 24 10:31:36.068: FEC: lacp_switch_add_port_to_associated_list_internal: Gi2/0/47 added to list for Po12
1342572: Aug 24 10:31:37.771: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342573: Aug 24 10:31:37.771: FEC: lacp_switch_get_first_associated_port_from_agg_id: found port Gi2/0/46 associated to Po12
1342574: Aug 24 10:31:37.771: FEC: lacp_switch_is_port_in_associate_list: port Gi2/0/46 is present in the associate list
1342575: Aug 24 10:31:37.771: FEC: lacp_switch_get_next_associated_port_from_agg_id: found port Gi2/0/47 next to Gi2/0/46 and associated to Po12
1342576: Aug 24 10:31:37.771: FEC: lacp_switch_is_port_in_associate_list: port Gi2/0/47 is present in the associate list
1342577: Aug 24 10:31:37.771: FEC: lacp_switch_get_next_associated_port_from_agg_id: no associated port next to Gi2/0/47 in aggregator Po12
1342578: Aug 24 10:31:37.771: FEC: lacp_switch_is_aggregator_valid: aggregator Po12 is still valid
1342579: Aug 24 10:31:37.771: FEC: lacp_switch_check_hw_sw_constraints_internal: port Gi2/0/47 can be bundled in the aggregator Po12, new afb->nports [1]

I have also tried the Cisco etherchannel mode with the FEC mode in OPNSense and the "channel-group 12 mode on" on catalyst : On the switch, ports get bundled together but I have no way to ping the LAN IP address.

Bye bye

[ThomasRicou]

Hi,
How may I get debug information on the FW ?
Thx

[mimugmail]

clog -f /var/log/system.log

On the Switch just ter mon and plug the cables