MAC Address Learning Issue ?!

[tryllz]

Hi All,

The network is as follows with VLAN interfaces on the firewall.

VLAN 15 - 10.10.15.1
VLAN 25 - 10.10.25.1
VLAN 26 - 10.10.26.1

NSX-T Edge Node Management - 10.10.15.101
NSX-T Edge Node Uplink 1 - 10.10.25.101
NSX-T Edge Node Uplink 2 - 10.10.26.102

I have OPNSense running as a VM on ESXi, and NSX-T Edge Node VM with 3 interfaces, Management, Uplink 1, Uplink 2.

I have Allowed Promiscuous Mode, MAC Address Changed, and Forged Transmits.

There are no firewall rules denying any traffic.

The problem is Edge Node Uplink 1 (10.10.25.101) can ping the firewall interface and vice versa, but interface 2 (10.10.26.102) cannot ping the firewall interface.

I did a packet capture on the firewall and the firewall interface 10.10.26.1 is sending ARP Broadcast when traceroute was performed from 10.10.26.102.





I have checked the ARP table and MAC address from Uplink 1 is added but Uplink 2 there are no entries from the Edge Node. I have set up another VM (10.10.26.225) on the 10.10.26.0 network and it can reach firewall interface (10.10.26.1) without any issues, and has entries from the VM (10.10.26.225) as well.



Anyone experienced with this issue, or knows what's going on, or what can be checked ?

Sorry been on this for about a week now.

[Patrick M. Hausen]

Misconfigured prefix length? /32 instead of /24 by accident?

[Saarbremer]

Did you check the configuration for uplink 2? Is it on the correct VLAN, is the prefix length correct?

Your opnSense asks for the mac address and gets no answer -> I doubt this is an opnsense issue.

[tryllz]

Both uplinks have the exact same configuration, and are in /24, and in the right VLAN as well.

Yesterday I did a firewall state table reset and the 2nd uplink started responding to ping normally, however, it still did not have any entry in the ARP Table, and upon reboot of the firewall and edge node the 2nd uplink went back to not responding to pings.

[Saarbremer]

I get confused.

A state reset enables pinging and a reboot stops it. That (at least in my cases so far) means usually a misconfiguration on network side.

I am confused with two things in your original post:

You claim that

NSX-T Edge Node Uplink 1 - 10.10.25.101
NSX-T Edge Node Uplink 2 - 10.10.26.102

but in the ARP table I see an entry for 10.10.25.102 and the same mac address for two IPs of your opnsense:
10.10.25.1
10.10.26.1


So is there 10.10.25.102 in your network, too?

[tryllz]

Hi,

Yes, there are 2 NSX-T Edges deployed, each with 2 Uplinks, and each uplink in a separate VLAN, its an Active/Active HA setup.

So :

Edge Node 1, Uplink 1 (10.10.25.101, VLAN 25), Uplink 2 (10.10.26.101, VLAN 26)

Edge Node 2, Uplink 1 (10.10.25.102, VLAN 25), Uplink 2 (10.10.26.102, VLAN 26)

Uplink 1 works with the firewall, while Uplink 2 does not.

10.10.25.1 and 10.10.26.1 are 2 VLAN interfaces that have the parent interface MAC address seen in the ARP table.

Ping from T0 VRF in Edge

Code Select Expand

edge1(tier0_sr[2])> ping 10.10.26.1
PING 10.10.26.1 (10.10.26.1): 56 data bytes
36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
4  5  00 0054 0000   0 0000  40  01 3230 10.10.26.101  10.10.26.1

Traceflow

[tryllz]

This is a test environment, using 1 firewall, and all the Edge Node uplinks are setup for BGP Peering on the same firewall.

[Saarbremer]

Ok, I am afraid that does not clear up things to me. However, could you paste the actual interface configuration from OPNSense for your interfaces, esp

VLAN 25 - 10.10.25.1
VLAN 26 - 10.10.26.1

Thanks

[tryllz]

Sorry about that.

I wasn't sure how to get interface information from CLI.

Network Overview

[Saarbremer]

I really don't see any problem here.

I guess you already checked the correct VLAN assignments on all involved (virtual and physical) network hardware. From the ARP protocol we still can tell that no ARP is returned.

Causes might be:
* Any ARP messages get filtered somewhere
* One of the ARP message directions is broken and packets are lost. (VLAN mismatch)

Given that you observed a change during state reset until the next proper reboot, I would check VLAN assignment on all involved devices and then check for any automatic or manual firewall rule on OPNsense (e.g. "block private networks" is off). The same for any possible filters involved on the ping target (if any).

Sorry for this generic advice. At least your interfaces, VLAN assignment & adressing scheme seem to be good on OPNsense.

[tryllz]

Appreciate you taking the time, and thanks for the feed back, definitely helpful in trying to narrow down the issue.

Will definitely recheck anything associated with VLANs.

[tryllz]

Seems like double tagging problem, 1 from vCenter Distributed Switch, and 2nd from OPNSense interface.

The issue is OPNSense VLAN interfaces cannot be created without tags, or cannot be set as 0 so tagging can be set at Distributed Switch level only.

I set the Edge Uplink portgroups to trunking.



And firewall ARP table now has the interface attached.



Now both interfaces are in Established state, and BGP peering on all Edge Interfaces successfully.

Code Select Expand

edge1> vrf 2
edge1(tier0_sr[2])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.101  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.10.25.1                          65555       Estab 00:12:58     UP  46      20      12     4
10.10.26.1                          65555       Estab 00:12:58     UP  46      20      12     14

Thu Aug 24 2023 UTC 17:54:55.772

Code Select Expand

edge2> vrf 1
edge2(tier0_sr[1])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
            AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.102  Local AS: 65000

Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx

10.10.25.1                          65555       Estab 00:15:18     UP  48      23      12     12
10.10.26.1                          65555       Estab 00:15:18     UP  51      23      12     6

Thu Aug 24 2023 UTC 17:57:02.232

[tryllz]

Any way to have VLAN interface without tagging at firewall ?

[Saarbremer]

Set VLAN untagged on the switch and omit VLAN tagging on OPNsense.

[tryllz]

Set VLAN untagged on the switch

This is possible, vCenter Distributed Switch can be configured.

omit VLAN tagging on OPNsense.

Sorry not sure I understand how to omit tagging on OPNSense. Do you mean by creating regular interfaces ?