LACP is not working

[Julien]

You positive you're plugged into the right ports?  FreeBSD is reporting a different mac address than your switch claims it's advertising on the LAG:

Nov 24 00:13:35 firewall kernel: partner=(0001,60-9C-9F-4B-80-8C,4E22,0001,0002)

Deployment: HW Trunk ID 2
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/2      Up      Blocked Full 1G    2     Yes N/A  0   609c.9f4b.808d LAN1
2/1/2      Up      Blocked Full 1G    2     Yes N/A  0   609c.9f4b.808d  LAN2

thank you for your answer, this probably because i created a new LAG to test it but still is not working.


I noticed a behaviour that was before at 19.1 on the lag interface https://github.com/opnsense/core/issues/3200

Code Select Expand

Nov 26 20:47:27 firewall opnsense[5775]: /usr/local/etc/rc.linkup: Hotplug event detected for LANLAG(opt1) but ignoring since interface is configured with static IP (192.168.88.1 ::)


contacted the support of brocade Ruck switch and they advised to disable strict mode on the lag.

I have disabled this mode using the below command but the LACP is not coming online
sysctl net.link.lagg.lacp.default_strict_mode=0

[Julien]

I am stil struggling with this.
now I am seeing a different logs on the opnsense. anyone a idea please?

Code Select Expand

Nov 28 17:44:32 firewall opnsense[19285]: /usr/local/etc/rc.linkup: Hotplug event detected for LANLAG(opt1) but ignoring since interface is configured with static IP (192.168.83.1 ::)
Nov 28 17:44:32 firewall opnsense[51767]: /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'lagg0'
Nov 28 17:44:32 firewall opnsense[51767]: /usr/local/etc/rc.newwanip: On (IP address: 192.168.83.1) (interface: LANLAG[opt1]) (real interface: lagg0).
Nov 28 17:44:32 firewall opnsense[51767]: plugins_configure hosts ()
Nov 28 17:44:32 firewall opnsense[51767]: plugins_configure hosts (execute task : dnsmasq_hosts_generate())
Nov 28 17:44:32 firewall opnsense[51767]: plugins_configure hosts (execute task : unbound_hosts_generate())
Nov 28 17:44:33 firewall kernel: bce1: Interface stopped DISTRIBUTING, possible flapping
Nov 28 17:44:33 firewall kernel: bce0: Interface stopped DISTRIBUTING, possible flapping
Nov 28 17:44:33 firewall kernel: lagg0: link state changed to DOWN
Nov 28 17:44:33 firewall opnsense[67168]: /usr/local/etc/rc.linkup: Hotplug event detected for LANLAG(opt1) but ignoring since interface is configured with static IP (192.168.83.1 ::)
Nov 28 17:44:39 firewall kernel: lagg0: link state changed to UP
Nov 28 17:44:40 firewall opnsense[99483]: /usr/local/etc/rc.linkup: Hotplug event detected for LANLAG(opt1) but ignoring since interface is configured with static IP (192.168.83.1 ::)
Nov 28 17:44:40 firewall opnsense[59439]: /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'lagg0'
Nov 28 17:44:40 firewall opnsense[59439]: /usr/local/etc/rc.newwanip: On (IP address: 192.168.83.1) (interface: LANLAG[opt1]) (real interface: lagg0).
Nov 28 17:44:40 firewall opnsense[59439]: plugins_configure hosts ()
Nov 28 17:44:40 firewall opnsense[59439]: plugins_configure hosts (execute task : dnsmasq_hosts_generate())
Nov 28 17:44:40 firewall opnsense[59439]: plugins_configure hosts (execute task : unbound_hosts_generate())
Nov 28 17:44:40 firewall kernel: bce1: Interface stopped DISTRIBUTING, possible flapping
Nov 28 17:44:40 firewall kernel: bce0: Interface stopped DISTRIBUTING, possible flapping
Nov 28 17:44:40 firewall kernel: lagg0: link state changed to DOWN
Nov 28 17:44:41 firewall opnsense[34568]: /usr/local/etc/rc.linkup: Hotplug event detected for LANLAG(opt1) but ignoring since interface is configured with static IP (192.168.83.1 ::)
Nov 28 17:44:42 firewall kernel: bce1: link state changed to DOWN
Nov 28 17:44:42 firewall kernel: bce0: link state changed to DOWN
Nov 28 17:44:43 firewall configctl[48202]: event @ 1606581883.04 msg: Nov 28 17:44:43 firewall.attcomputer.nl config[96814]: config-event: new_config /conf/backup/config-1606581883.0416.xml
Nov 28 17:44:43 firewall configctl[48202]: event @ 1606581883.04 exec: system event config_changed
Nov 28 17:44:45 firewall kernel: bce1: Gigabit link up!
Nov 28 17:44:45 firewall kernel: bce1: link state changed to UP
Nov 28 17:44:45 firewall kernel: lagg0: link state changed to UP
Nov 28 17:44:45 firewall kernel: bce0: Gigabit link up!
Nov 28 17:44:45 firewall kernel: bce0: link state changed to UP
Nov 28 17:44:45 firewall opnsense[49000]: /usr/local/etc/rc.linkup: Hotplug event detected for LANLAG(opt1) but ignoring since interface is configured with static IP (192.168.83.1 ::)
Nov 28 17:44:45 firewall opnsense[38393]: /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'lagg0'
Nov 28 17:44:45 firewall opnsense[38393]: /usr/local/etc/rc.newwanip: On (IP address: 192.168.83.1) (interface: LANLAG[opt1]) (real interface: lagg0).
Nov 28 17:44:45 firewall opnsense[38393]: plugins_configure hosts ()
Nov 28 17:44:45 firewall opnsense[38393]: plugins_configure hosts (execute task : dnsmasq_hosts_generate())
Nov 28 17:44:45 firewall opnsense[38393]: plugins_configure hosts (execute task : unbound_hosts_generate())
Nov 28 17:45:01 firewall /update_tables.py[249]: unable to resolve firewall-new-wa.nl for alias Klante_S2S
Nov 28 17:45:12 firewall /flowd_aggregate.py[18361]: vacuum src_addr_details_086400.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum src_addr_000300.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum src_addr_003600.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum src_addr_086400.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum dst_port_000300.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum dst_port_003600.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum dst_port_086400.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum interface_000030.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum interface_000300.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum interface_003600.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum interface_086400.sqlite
Nov 28 17:45:13 firewall /flowd_aggregate.py[18361]: vacuum done

[Gauss23]

Did not read the full post.
Maybe it's a BSD12 topic. Did you try it with pfSense 2.5?
Is it running with OPNsense 20.1?

[Julien]

Did not read the full post.
Maybe it's a BSD12 topic. Did you try it with pfSense 2.5?
Is it running with OPNsense 20.1?

i havent tried it to be honest, with pfsense is working fine.
i am worried on getting it working now, and later it will maybe crashes if update will change the LACP behaivor.
i am at 20.7 i cannot go back to 20.1 ?

[mimugmail]

Reinstall 20.1 and restore config

[Julien]

Reinstall 20.1 and restore config

thank you for your answer.
i am not near the box have to drive 2hr to get there.
is this a bug/issue with 20.7?
just trying to understand to plan my next move.


i have fired up a hardware home with unifi switch, i ve created LACP and it seems to work.

this working with this release

Code Select Expand

OPNsense 20.7.5-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1h 22 Sep 2020

Code Select Expand

Nov 28 21:52:08 firewall kernel: igb5: lacpdu transmit
Nov 28 21:52:08 firewall kernel: actor=(8000,00-08-A2-0C-99-7B,020B,8000,0006)
Nov 28 21:52:08 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:08 firewall kernel: partner=(8000,74-83-C2-48-2F-67,0042,0080,0018)
Nov 28 21:52:08 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:08 firewall kernel: maxdelay=0
Nov 28 21:52:08 firewall kernel: igb4: lacpdu transmit
Nov 28 21:52:08 firewall kernel: actor=(8000,00-08-A2-0C-99-7B,020B,8000,0005)
Nov 28 21:52:08 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:08 firewall kernel: partner=(8000,74-83-C2-48-2F-67,0042,0080,0017)
Nov 28 21:52:08 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:08 firewall kernel: maxdelay=0
Nov 28 21:52:16 firewall kernel: igb5: lacpdu receive
Nov 28 21:52:16 firewall kernel: actor=(8000,74-83-C2-48-2F-67,0042,0080,0018)
Nov 28 21:52:16 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:16 firewall kernel: partner=(8000,00-08-A2-0C-99-7B,020B,8000,0006)
Nov 28 21:52:16 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:16 firewall kernel: maxdelay=0
Nov 28 21:52:16 firewall kernel: igb4: lacpdu receive
Nov 28 21:52:16 firewall kernel: actor=(8000,74-83-C2-48-2F-67,0042,0080,0017)
Nov 28 21:52:16 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:16 firewall kernel: partner=(8000,00-08-A2-0C-99-7B,020B,8000,0005)
Nov 28 21:52:16 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:16 firewall kernel: maxdelay=0
Nov 28 21:52:38 firewall kernel: igb5: lacpdu transmit
Nov 28 21:52:38 firewall kernel: actor=(8000,00-08-A2-0C-99-7B,020B,8000,0006)
Nov 28 21:52:38 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:38 firewall kernel: partner=(8000,74-83-C2-48-2F-67,0042,0080,0018)
Nov 28 21:52:38 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:38 firewall kernel: maxdelay=0
Nov 28 21:52:38 firewall kernel: igb4: lacpdu transmit
Nov 28 21:52:38 firewall kernel: actor=(8000,00-08-A2-0C-99-7B,020B,8000,0005)
Nov 28 21:52:38 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:38 firewall kernel: partner=(8000,74-83-C2-48-2F-67,0042,0080,0017)
Nov 28 21:52:38 firewall kernel:

[Gauss23]

thank you for your answer.
i am not near the box have to drive 2hr to get there.
is this a bug/issue with 20.7?
just trying to understand to plan my next move.

It may be related to BSD12. So even if you go for pfSense 2.4.x now, it may break with 2.5.
That's the reason why I asked if it's working with OPNsense 20.1 or pfSense 2.5

It could be a combination of BSD12 and your switches, because other people say it's working for them.

Just found this:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241785

Is this describing your issue?

[Julien]

thank you for your answer.
i am not near the box have to drive 2hr to get there.
is this a bug/issue with 20.7?
just trying to understand to plan my next move.

It may be related to BSD12. So even if you go for pfSense 2.4.x now, it may break with 2.5.
That's the reason why I asked if it's working with OPNsense 20.1 or pfSense 2.5

It could be a combination of BSD12 and your switches, because other people say it's working for them.

Just found this:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241785

Is this describing your issue?

thank you for your answer.
yes its my issue. exactly as showen.
where do they mean to disable those settings?

Code Select Expand

We also use vlan + lagg + ix and we often need to add/remove vlans, so as a temporary solution we disable vlanhwfilter on lagg interface.

i just tried it nowhome with a unifi switch and it works. same settings.

Code Select Expand

OPNsense 20.7.5-amd64
FreeBSD 12.1-RELEASE-p10-HBSD
OpenSSL 1.1.1h 22 Sep 2020

Code Select Expand

Nov 28 21:52:08 firewall kernel: igb5: lacpdu transmit
Nov 28 21:52:08 firewall kernel: actor=(8000,00-08-A2-0C-99-7B,020B,8000,0006)
Nov 28 21:52:08 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:08 firewall kernel: partner=(8000,74-83-C2-48-2F-67,0042,0080,0018)
Nov 28 21:52:08 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:08 firewall kernel: maxdelay=0
Nov 28 21:52:08 firewall kernel: igb4: lacpdu transmit
Nov 28 21:52:08 firewall kernel: actor=(8000,00-08-A2-0C-99-7B,020B,8000,0005)
Nov 28 21:52:08 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:08 firewall kernel: partner=(8000,74-83-C2-48-2F-67,0042,0080,0017)
Nov 28 21:52:08 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:08 firewall kernel: maxdelay=0
Nov 28 21:52:16 firewall kernel: igb5: lacpdu receive
Nov 28 21:52:16 firewall kernel: actor=(8000,74-83-C2-48-2F-67,0042,0080,0018)
Nov 28 21:52:16 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:16 firewall kernel: partner=(8000,00-08-A2-0C-99-7B,020B,8000,0006)
Nov 28 21:52:16 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:16 firewall kernel: maxdelay=0
Nov 28 21:52:16 firewall kernel: igb4: lacpdu receive
Nov 28 21:52:16 firewall kernel: actor=(8000,74-83-C2-48-2F-67,0042,0080,0017)
Nov 28 21:52:16 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:16 firewall kernel: partner=(8000,00-08-A2-0C-99-7B,020B,8000,0005)
Nov 28 21:52:16 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:16 firewall kernel: maxdelay=0
Nov 28 21:52:38 firewall kernel: igb5: lacpdu transmit
Nov 28 21:52:38 firewall kernel: actor=(8000,00-08-A2-0C-99-7B,020B,8000,0006)
Nov 28 21:52:38 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:38 firewall kernel: partner=(8000,74-83-C2-48-2F-67,0042,0080,0018)
Nov 28 21:52:38 firewall kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:38 firewall kernel: maxdelay=0
Nov 28 21:52:38 firewall kernel: igb4: lacpdu transmit
Nov 28 21:52:38 firewall kernel: actor=(8000,00-08-A2-0C-99-7B,020B,8000,0005)
Nov 28 21:52:38 firewall kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Nov 28 21:52:38 firewall kernel: partner=(8000,74-83-C2-48-2F-67,0042,0080,0017)
Nov 28 21:52:38 firewall kernel:

[Gauss23]

Interfaces: Settings

VLAN Hardware Filtering. Set it to disable.

And all other HW stuff should be disabled (checkboxes checked), too.

It's a global setting for all interfaces.

[Julien]

Interfaces: Settings

VLAN Hardware Filtering. Set it to disable.

And all other HW stuff should be disabled (checkboxes checked), too.

It's a global setting for all interfaces.

thank you, i just disabled it now.
it was enabled : Enable VLAN hardware Filtering now its disabled.
ill see if i can get the LACP working now
will report back in a min


edit: unfrotunately its still not working

Code Select Expand

lagg0: link state changed to UP
bce1: Interface stopped DISTRIBUTING, possible flapping
bce0: Interface stopped DISTRIBUTING, possible flapping
lagg0: link state changed to DOWN
lagg0: link state changed to UP
bce1: Interface stopped DISTRIBUTING, possible flapping
bce0: Interface stopped DISTRIBUTING, possible flapping
lagg0: link state changed to DOWN
lagg0: link state changed to UP
bce1: Interface stopped DISTRIBUTING, possible flapping
bce0: Interface stopped DISTRIBUTING, possible flapping
lagg0: link state changed to DOWN
lagg0: link state changed to UP
bce1: Interface stopped DISTRIBUTING, possible flapping
bce0: Interface stopped DISTRIBUTING, possible flapping
lagg0: link state changed to DOWN

[Gauss23]

edit: unfrotunately its still not working

I'm really sorry to hear that.

Options:

• go with pfSense 2.4.x but you may run into the problem again when 2.5 is released. So maybe it's worth to try the 2.5 beta first. If it has the same problem it's BSD 12 related.
  
• try OPNsense 20.1 and wait if there will be a patch someday
  
• try different switches
  
• are you able to replace the network cards in that server? Maybe it's only related to some drivers.
  

[Julien]

edit: unfrotunately its still not working

I'm really sorry to hear that.

Options:

• go with pfSense 2.4.x but you may run into the problem again when 2.5 is released. So maybe it's worth to try the 2.5 beta first. If it has the same problem it's BSD 12 related.
  
• try OPNsense 20.1 and wait if there will be a patch someday
  
• try different switches
  
• are you able to replace the network cards in that server? Maybe it's only related to some drivers.
  

thank you so much for your continue answers. i'll reach out to the support and see what is the cause.

the server has two different NICS i tried them both same error.

appreciate it really


Edit.

i noticed two LAGS are sharing the same MAC address is this maybe the cause?

Code Select Expand


Deployment: HW Trunk ID 1
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1      Down    None    None None  1     No  141  0   609c.9f4b.808c  WAN1
2/1/1      Down    None    None None  1     No  141  0   609c.9f4b.808c  WAN2

Code Select Expand

Deployment: HW Trunk ID 3
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/11     Up      Forward Full 1G    11    Yes 141  0   609c.9f4b.808c
1/1/12     Up      Forward Full 1G    11    Yes 141  0   609c.9f4b.808c
2/1/11     Up      Forward Full 1G    11    Yes 141  0   609c.9f4b.808c
2/1/12     Up      Forward Full 1G    11    Yes 141  0   609c.9f4b.808c

[djbmister]

To help along further with this, there is some possible issues occuring:

1. Have you turned of all offloading options on opnsense? - TSO, LRO? - make sure with the ifconfig
2. Port flap dampening configuration is worth looking at on this switch?
3. Energy saving features of the nics on the server - i.e. hw.em.eee_setting = 0

Maybe do a 'sysctl -A | grep *your network card driver* - 'sysctl -A | grep em or igb'

Also sysctl -A | grep lacp - lets see your lacp settings

[Julien]

To help along further with this, there is some possible issues occuring:

1. Have you turned of all offloading options on opnsense? - TSO, LRO? - make sure with the ifconfig
2. Port flap dampening configuration is worth looking at on this switch?
3. Energy saving features of the nics on the server - i.e. hw.em.eee_setting = 0

Maybe do a 'sysctl -A | grep *your network card driver* - 'sysctl -A | grep em or igb'

Also sysctl -A | grep lacp - lets see your lacp settings

Thank you for your answer

Code Select Expand

1. Have you turned of all offloading options on opnsense? - TSO, LRO? - make sure with the ifconfig
yes i did in interface settings , Hardware CRC / Hardware TSO /Hardware LRO / VLAN Hardware Filtering    are disabled
when i do ifconfig -vv lagg0

Code Select Expand

lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800008<VLAN_MTU>
        ether e8:39:35:11:fa:ab
        inet6 fe80::ea39:35ff:fe11:faab%lagg0 prefixlen 64 scopeid 0xb
        laggproto lacp lagghash l2,l3,l4
        lagg options:
                flags=10<LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 0
                flapping: 0
        lag id: [(0000,00-00-00-00-00-00,0000,0000,0000),
                 (0000,00-00-00-00-00-00,0000,0000,0000)]
        laggport: em2 flags=0<> state=41<ACTIVITY,DEFAULTED>
                [(8000,E8-39-35-11-FA-AB,8003,8000,0003),
                 (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
        laggport: em3 flags=0<> state=41<ACTIVITY,DEFAULTED>
                [(8000,E8-39-35-11-FA-AB,8004,8000,0004),
                 (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
        groups: lagg
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Code Select Expand

2. Port flap dampening configuration is worth looking at on this switch?
link error dampening is enabled

Code Select Expand

SSH@Ruckus2050-LS#show interface ethernet 1/1/2
GigabitEthernet1/1/2 is up, line protocol is down (LACP-BLOCKED)
  Port down (LACP-BLOCKED) for 1 day(s) 14 hour(s) 28 minute(s) 40 second(s)
  Hardware is GigabitEthernet, address is 609c.9f4b.808d (bia 609c.9f4b.808d)
  Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual MDIX
  EEE Feature Disabled
  Member of 7 L2 VLANs, port is tagged, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Enabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Member of active trunk ports 1/1/2,2/1/2, primary port is 1/1/2
  Member of configured trunk ports 1/1/2,2/1/2, primary port is 1/1/2
  Port name is LAN1
  IPG MII 96 bits-time, IPG GMII 96 bits-time
  MTU 10200 bytes, encapsulation ethernet
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 928 bits/sec, 0 packets/sec, 0.00% utilization
  15187 packets input, 1943872 bytes, 0 no buffer
  Received 1 broadcasts, 15186 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  154231 packets output, 19755504 bytes, 0 underruns
  Transmitted 214 broadcasts, 153930 multicasts, 86 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

UC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                   0                   0
         1                   0                   0
         2                   0                   0
         3                   0                   0
         4                   0                   0
         5                   0                   0
         6                   0                   0
         7              152356                   0


MC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                 217                   0
         1                 148                   0
         2                1510                   0
         3                   0                   0

Code Select Expand

3. Energy saving features of the nics on the server - i.e. hw.em.eee_setting = 0

i am not sure i understand this correctly?

I see also MAC- learning is enabled, we have two opnsense boxes connected to the switch one is on and one is off, both boxes run the same configuration in case first one goes down we fired up the second one, maybe the Mac-learning causing this?

[djbmister]

Could you run

'sysctl -A | grep *your network card driver* - i.e. 'sysctl -A | grep em or igb'

Also 'sysctl -A | grep lacp' - lets see your lacp settings

And post output of each command, em or igb is the intel driver

eee is an energy efficient feature can cause issues on freebsd for nics. By setting 'hw.em.eee_setting = 0' in the tunables will turn this off for all nics.

Also what is your lagg settings on your opnsense? - have you tried loadbalance mode?

also on opnsense, lets see the lacp debugging

'sysctl net.link.lagg.lacp.debug=1' - then share the system log - 'clog /var/log/system.log'


IMAHO: It seems someone else has the same issue as you on pfsense - https://forum.netgate.com/topic/158534/lacp-not-working/79 - and thats a brocade switch.