Hello,
I need assistance, I'm 70% sure about this is related to the OPNsense configuration, or maybe 30% it's a VLAN config issue.
I'm running
- OPNsense 23.1
- 1 x Quad 1GB network interface
- 1 x Fiber SPF+ fiber to copper with Cat8 cable
- Cisco C2960
Here is my network setup :
(igb0) (Gi1/0/1)
(wan) |¯¯¯¯¯¯¯¯¯¯| (ix0-10gb) |¯¯¯¯¯¯¯¯¯¯|-------------------|¯¯¯¯¯¯¯¯| (Gi1/0/13) |¯¯¯¯¯¯¯|
WAN ------------|ISP ROUTER|-------------| OPNSENSE | LACP | C2960 |--------------| PC |
|__________| |__________|-------------------|________| |_______|
(igb1) (Gi1/0/2)
|¯¯¯¯¯¯¯¯¯¯| |¯¯¯¯¯¯¯¯¯¯¯¯¯¯|-------------------|¯¯¯¯¯¯¯¯¯¯¯¯¯¯| (10.0.150.0/24) |¯¯¯¯¯¯¯¯¯¯¯|
WAN --------|ISP ROUTER|-------------| OPNSENSE (.1)| | C2960 (.254) |--------------------| PC (.10) |
|__________| |______________|-------------------|______________| |___________|
My C2960 config look like this :
!
interface Port-channel1
description opnsense link aggregation
switchport trunk allowed vlan 150
switchport mode trunk
!
interface GigabitEthernet1/0/1
switchport trunk allowed vlan 150
switchport mode trunk
channel-group 1 mode active
!
interface GigabitEthernet1/0/2
switchport trunk allowed vlan 150
switchport mode trunk
channel-group 1 mode active
!
!
!
!
interface GigabitEthernet1/0/13
switchport access vlan 150
!
!
!
interface Vlan150
description vlan150
ip address 10.0.150.254 255.255.255.0
!
My output from my LACP
SW2960# show lacp neighbor
Flags: S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode
Channel group 1 neighbors
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi1/0/1 FA 32768 1111.5f15.2222 29s 0x0 0x16B 0x1 0x3F
Gi1/0/2 FA 32768 1111.5f15.2222 29s 0x0 0x16B 0x2 0x3F
SW2960#
My OPNsense config look like this -- see attachment.
Troubleshooting
- C2960 and OPNsense don't see their MAC
- PC (10.0.150.5 is able to ping Vlan150 at 10.0.150.254 on the C2960.
- PC don't see the OPNsense MAC
What I'm missing? :o
Any idea? ;D
Thanks
First thought (but could be wrong) Firewall Rules.
What does the layer 2 VLAN configuration look like? See attached screenshot for the settings I refer to.
Quote from: iammike on June 27, 2023, 09:14:44 AM
First thought (but could be wrong) Firewall Rules.
ARP entry must be visible at this layer.
Quote from: pmhausen on June 27, 2023, 09:17:56 AM
What does the layer 2 VLAN configuration look like? See attached screenshot for the settings I refer to.
Same configuration (see attached screenshot)
What is your OPNsense plugged into?
Do you have an LACP config to a Cisco switch? If so, is the config also similar?
Yes, Cisco 2960-L, works perfectly. Configuration identical. I would first remove the "allowed vlans" statement just to be sure. Also check if the PC is really connected to an access port assigned VLAN 150 on the Cisco side.
"ifconfig -v lagg0" will show you the LACP state as OPNsense sees it.
Quote from: pmhausen on June 27, 2023, 01:44:24 PM
Yes, Cisco 2960-L, works perfectly. Configuration identical. I would first remove the "allowed vlans" statement just to be sure. Also check if the PC is really connected to an access port assigned VLAN 150 on the Cisco side.
"ifconfig -v lagg0" will show you the LACP state as OPNsense sees it.
Hi pmhausen,
Here is the output from "ifconfig -v lagg0"
root@gw:~ # ifconfig -v lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: OPT4 (opt4)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 80:61:5f:15:a4:67
laggproto lacp lagghash l2,l3,l4
lagg options:
flags=14<USE_NUMA,LACP_STRICT>
flowid_shift: 16
lagg statistics:
active ports: 2
flapping: 0
lag id: [(8000,80-61-5F-15-A4-67,016B,0000,0000),
(8000,DC-CE-C1-CB-59-80,0001,0000,0000)]
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
[(8000,80-61-5F-15-A4-67,016B,8000,0001),
(8000,DC-CE-C1-CB-59-80,0001,8000,0102)]
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
[(8000,80-61-5F-15-A4-67,016B,8000,0002),
(8000,DC-CE-C1-CB-59-80,0001,8000,0103)]
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@gw:~ #
After many attempt, I'm able to isolate the issue to the OPNsense config.
I started a new VLAN 200 interface to start from scratch.
OPNsense result :
ping OPNsense TO OPNsense --> success
root@gw:~ #
root@gw:~ # ping 10.0.200.1
PING 10.0.200.1 (10.0.200.1): 56 data bytes
64 bytes from 10.0.200.1: icmp_seq=0 ttl=64 time=0.049 ms
64 bytes from 10.0.200.1: icmp_seq=1 ttl=64 time=0.039 ms
64 bytes from 10.0.200.1: icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from 10.0.200.1: icmp_seq=3 ttl=64 time=0.043 ms
^C
--- 10.0.200.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.039/0.044/0.049/0.003 ms
root@gw:~ #
ping OPNsense TO C2960 --> failed
root@gw:~ #
root@gw:~ # ping 10.0.200.254
PING 10.0.200.254 (10.0.200.254): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.0.200.254 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@gw:~ #
I started a new config LAGG/LACP with a new VLAN
Result : I'm able to ping the C2960 from the Gi1/0/25 using my PC.
But the OPNsense is still having "network down issue"
interface Port-channel1
description opnsense link aggregation
switchport trunk allowed vlan 125,200
switchport mode trunk
!
interface GigabitEthernet1/0/1
switchport trunk allowed vlan 125,200
switchport mode trunk
channel-group 1 mode active
!
interface GigabitEthernet1/0/2
switchport trunk allowed vlan 125,200
switchport mode trunk
channel-group 1 mode active
!
interface GigabitEthernet1/0/25
description PC
switchport access vlan 200
!
interface Vlan200
ip address 10.0.200.254 255.255.255.0
!
sw.local#show int vlan200
Vlan200 is up, line protocol is up
Change the lagghash to l2,l3 on the OPNsense side ... Cisco does not do l4.
Quote from: pmhausen on June 28, 2023, 08:42:53 AM
Change the lagghash to l2,l3 on the OPNsense side ... Cisco does not do l4.
Tested, and not working.
I selected L2 + L3 in the LAGG config.
But check that.. I'm able to ping/see ARP for 10.0.100.254 (C2960 VLAN 100) but not ping/see ARP entry for 10.0.200.254 (C2960 VLAN 200).
root@gw:~ # arp -a
? (10.0.200.1) at 00:00:00:00:00:00 on vlan02 permanent [vlan]
gw.sd.local (10.0.100.1) at 80:61:5f:15:a4:6a on vlan01 permanent [vlan]
root@gw:~ # ping 10.0.200.254
PING 10.0.200.254 (10.0.200.254): 56 data bytes
ping: sendto: Network is down
^C
--- 10.0.200.254 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
root@gw:~ # ping 10.0.200.254
PING 10.0.200.254 (10.0.200.254): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.0.200.254 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss
root@gw:~ #
And check this from the OPNsense dashboard, I don't see the MAC address either on the VLAN200 interface -- see attachment.
Thank you very much for your help
Did you actually create the VLAN on the Cisco?
If yes I would need an ifconfig -a on the OPNsense and a complete show run from the Cisco minus any credentials/passwords. Otherwise I don't see anything wrong.
Just ran into the same/similar issue i.e. trying to pass VLAN tagged traffic over an LACP trunk to a Cisco 2960-S. The solution for me was to set the system MTU to 9000
conf t
system mtu jumbo 9000
A switch reboot is needed.
Quote from: duka9 on June 28, 2023, 05:43:59 AM
Here is the output from "ifconfig -v lagg0"
root@gw:~ # ifconfig -v lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: OPT4 (opt4)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
ether 80:61:5f:15:a4:67
laggproto lacp lagghash l2,l3,l4
lagg options:
flags=14<USE_NUMA,LACP_STRICT>
flowid_shift: 16
lagg statistics:
active ports: 2
flapping: 0
lag id: [(8000,80-61-5F-15-A4-67,016B,0000,0000),
(8000,DC-CE-C1-CB-59-80,0001,0000,0000)]
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
[(8000,80-61-5F-15-A4-67,016B,8000,0001),
(8000,DC-CE-C1-CB-59-80,0001,8000,0102)]
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
[(8000,80-61-5F-15-A4-67,016B,8000,0002),
(8000,DC-CE-C1-CB-59-80,0001,8000,0103)]
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root@gw:~ #
Did you set the LACP Fast option at the OPNsense side ? Edit: INTERFACES: OTHER TYPES: LAGG
So the output (ifconfig -v lagg0) shows something like this:
lagg options:
flags=80<LACP_FAST_TIMO>
flowid_shift: 16
Because your Cisco Switch is configured with it:
SW2960# show lacp neighbor
Flags: S - Device is requesting Slow LACPDUs
F - Device is requesting Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode
Channel group 1 neighbors
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi1/0/1 FA 32768 1111.5f15.2222 29s 0x0 0x16B 0x1 0x3F
Gi1/0/2 FA 32768 1111.5f15.2222 29s 0x0 0x16B 0x2 0x3F
SW2960#