Question about LACP between OPNsense with Cisco 2960

[duka9]

Hello,

I need assistance, I'm 70% sure about this is related to the OPNsense configuration, or maybe 30% it's a VLAN config issue.

I'm running

- OPNsense 23.1
   - 1 x Quad 1GB network interface
   - 1 x Fiber SPF+ fiber to copper with Cat8 cable
- Cisco C2960

Here is my network  setup :

Code: [Select]



      (igb0)   (Gi1/0/1)    
(wan) |¯¯¯¯¯¯¯¯¯¯|  (ix0-10gb) |¯¯¯¯¯¯¯¯¯¯|-------------------|¯¯¯¯¯¯¯¯| (Gi1/0/13) |¯¯¯¯¯¯¯|
WAN ------------|ISP ROUTER|-------------| OPNSENSE |  LACP | C2960  |--------------|  PC |
|__________| |__________|-------------------|________| |_______|
      (igb1)   (Gi1/0/2)


Code: [Select]

    |¯¯¯¯¯¯¯¯¯¯|             |¯¯¯¯¯¯¯¯¯¯¯¯¯¯|-------------------|¯¯¯¯¯¯¯¯¯¯¯¯¯¯|  (10.0.150.0/24)   |¯¯¯¯¯¯¯¯¯¯¯|
WAN --------|ISP ROUTER|-------------| OPNSENSE (.1)|         | C2960 (.254) |--------------------| PC (.10)  |
    |__________|      |______________|-------------------|______________|     |___________|


My C2960 config look like this :

Code: [Select]

!
interface Port-channel1
description opnsense link aggregation
switchport trunk allowed vlan 150
switchport mode trunk
!
interface GigabitEthernet1/0/1
switchport trunk allowed vlan 150
switchport mode trunk
channel-group 1 mode active
!
interface GigabitEthernet1/0/2
switchport trunk allowed vlan 150
switchport mode trunk
channel-group 1 mode active
!
!
!
!
interface GigabitEthernet1/0/13
switchport access vlan 150
!
!
!
interface Vlan150
description vlan150
ip address 10.0.150.254 255.255.255.0
!

My output from my LACP

Code: [Select]


        SW2960# show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

  LACP port                        Admin  Oper   Port    Port
Port      Flags   Priority  Dev ID          Age    key    Key    Number  State
Gi1/0/1   FA      32768     1111.5f15.2222  29s    0x0    0x16B  0x1     0x3F
Gi1/0/2   FA      32768     1111.5f15.2222  29s    0x0    0x16B  0x2     0x3F
SW2960#


My OPNsense config look like this -- see attachment.

Troubleshooting

   - C2960 and OPNsense don't see their MAC
   - PC (10.0.150.5 is able to ping Vlan150 at 10.0.150.254 on the C2960.
        - PC don't see the OPNsense MAC

What I'm missing? 

Any idea?

Thanks

[iammike]

First thought (but could be wrong) Firewall Rules.

[Patrick M. Hausen]

What does the layer 2 VLAN configuration look like? See attached screenshot for the settings I refer to.

[duka9]

First thought (but could be wrong) Firewall Rules.

ARP entry must be visible at this layer.

[duka9]

What does the layer 2 VLAN configuration look like? See attached screenshot for the settings I refer to.

Same configuration (see attached screenshot)

What is your OPNsense plugged into?
Do you have an LACP config to a Cisco switch? If so, is the config also similar?

[Patrick M. Hausen]

Yes, Cisco 2960-L, works perfectly. Configuration identical. I would first remove the "allowed vlans" statement just to be sure. Also check if the PC is really connected to an access port assigned VLAN 150 on the Cisco side.

"ifconfig -v lagg0" will show you the LACP state as OPNsense sees it.

[duka9]

Yes, Cisco 2960-L, works perfectly. Configuration identical. I would first remove the "allowed vlans" statement just to be sure. Also check if the PC is really connected to an access port assigned VLAN 150 on the Cisco side.

"ifconfig -v lagg0" will show you the LACP state as OPNsense sees it.

Hi pmhausen,

Here is the output from "ifconfig -v lagg0"

Code: [Select]

root@gw:~ # ifconfig -v lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT4 (opt4)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether 80:61:5f:15:a4:67
        laggproto lacp lagghash l2,l3,l4
        lagg options:
                flags=14<USE_NUMA,LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 2
                flapping: 0
        lag id: [(8000,80-61-5F-15-A4-67,016B,0000,0000),
                 (8000,DC-CE-C1-CB-59-80,0001,0000,0000)]
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,80-61-5F-15-A4-67,016B,8000,0001),
                 (8000,DC-CE-C1-CB-59-80,0001,8000,0102)]
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,80-61-5F-15-A4-67,016B,8000,0002),
                 (8000,DC-CE-C1-CB-59-80,0001,8000,0103)]
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@gw:~ #

After many attempt, I'm able to isolate the issue to the OPNsense config.

I started a new VLAN 200 interface to start from scratch.

OPNsense result :

ping OPNsense TO OPNsense --> success

Code: [Select]

root@gw:~ #
root@gw:~ # ping 10.0.200.1
PING 10.0.200.1 (10.0.200.1): 56 data bytes
64 bytes from 10.0.200.1: icmp_seq=0 ttl=64 time=0.049 ms
64 bytes from 10.0.200.1: icmp_seq=1 ttl=64 time=0.039 ms
64 bytes from 10.0.200.1: icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from 10.0.200.1: icmp_seq=3 ttl=64 time=0.043 ms
^C
--- 10.0.200.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.039/0.044/0.049/0.003 ms
root@gw:~ #

ping OPNsense TO C2960 --> failed

Code: [Select]

root@gw:~ #
root@gw:~ # ping 10.0.200.254
PING 10.0.200.254 (10.0.200.254): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.0.200.254 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@gw:~ #

I started a new config LAGG/LACP with a new VLAN

Result : I'm able to ping the C2960 from the Gi1/0/25 using my PC.

But the OPNsense is still having "network down issue"

Code: [Select]

interface Port-channel1
 description opnsense link aggregation
 switchport trunk allowed vlan 125,200
 switchport mode trunk
!
interface GigabitEthernet1/0/1
 switchport trunk allowed vlan 125,200
 switchport mode trunk
 channel-group 1 mode active
!
interface GigabitEthernet1/0/2
 switchport trunk allowed vlan 125,200
 switchport mode trunk
 channel-group 1 mode active
!

interface GigabitEthernet1/0/25
 description PC
 switchport access vlan 200
!
interface Vlan200
 ip address 10.0.200.254 255.255.255.0
!

sw.local#show int vlan200
Vlan200 is up, line protocol is up


[Patrick M. Hausen]

Change the lagghash to l2,l3 on the OPNsense side ... Cisco does not do l4.

[duka9]

Change the lagghash to l2,l3 on the OPNsense side ... Cisco does not do l4.

Tested, and not working.

I selected L2 + L3 in the LAGG config.

But check that.. I'm able to ping/see ARP for 10.0.100.254 (C2960 VLAN 100) but not ping/see ARP entry for 10.0.200.254 (C2960 VLAN 200).

Code: [Select]

root@gw:~ # arp -a
? (10.0.200.1) at 00:00:00:00:00:00 on vlan02 permanent [vlan]
gw.sd.local (10.0.100.1) at 80:61:5f:15:a4:6a on vlan01 permanent [vlan]

root@gw:~ # ping 10.0.200.254
PING 10.0.200.254 (10.0.200.254): 56 data bytes
ping: sendto: Network is down
^C
--- 10.0.200.254 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
root@gw:~ # ping 10.0.200.254
PING 10.0.200.254 (10.0.200.254): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.0.200.254 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss
root@gw:~ #

And check this from the OPNsense dashboard, I don't see the MAC address either on the VLAN200 interface -- see attachment.

Thank you very much for your help

[Patrick M. Hausen]

Did you actually create the VLAN on the Cisco?

If yes I would need an ifconfig -a on the OPNsense and a complete show run from the Cisco minus any credentials/passwords. Otherwise I don't see anything wrong.

[Manxmann]

Just ran into the same/similar issue i.e. trying to pass VLAN tagged traffic over an LACP trunk to a Cisco 2960-S. The solution for me was to set the system MTU to 9000

conf t
system mtu jumbo 9000

A switch reboot is needed.

[netnut]

Here is the output from "ifconfig -v lagg0"

Code: [Select]

root@gw:~ # ifconfig -v lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: OPT4 (opt4)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether 80:61:5f:15:a4:67
        laggproto lacp lagghash l2,l3,l4
        lagg options:
                flags=14<USE_NUMA,LACP_STRICT>
                flowid_shift: 16
        lagg statistics:
                active ports: 2
                flapping: 0
        lag id: [(8000,80-61-5F-15-A4-67,016B,0000,0000),
                 (8000,DC-CE-C1-CB-59-80,0001,0000,0000)]
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,80-61-5F-15-A4-67,016B,8000,0001),
                 (8000,DC-CE-C1-CB-59-80,0001,8000,0102)]
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
                [(8000,80-61-5F-15-A4-67,016B,8000,0002),
                 (8000,DC-CE-C1-CB-59-80,0001,8000,0103)]
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@gw:~ #


Did you set the LACP Fast option at the OPNsense side ? Edit:  INTERFACES: OTHER TYPES: LAGG

So the output (ifconfig -v lagg0) shows something like this:

Code: [Select]

lagg options:
flags=80<LACP_FAST_TIMO>
flowid_shift: 16

Because your Cisco Switch is configured with it:

Code: [Select]

SW2960# show lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

  LACP port                        Admin  Oper   Port    Port
Port      Flags   Priority  Dev ID          Age    key    Key    Number  State
Gi1/0/1   FA      32768     1111.5f15.2222  29s    0x0    0x16B  0x1     0x3F
Gi1/0/2   FA      32768     1111.5f15.2222  29s    0x0    0x16B  0x2     0x3F
SW2960#