Hi All,
The network is as follows with VLAN interfaces on the firewall.
VLAN 15 - 10.10.15.1
VLAN 25 - 10.10.25.1
VLAN 26 - 10.10.26.1
NSX-T Edge Node Management - 10.10.15.101
NSX-T Edge Node Uplink 1 - 10.10.25.101
NSX-T Edge Node Uplink 2 - 10.10.26.102
I have OPNSense running as a VM on ESXi, and NSX-T Edge Node VM with 3 interfaces, Management, Uplink 1, Uplink 2.
I have Allowed Promiscuous Mode, MAC Address Changed, and Forged Transmits.
There are no firewall rules denying any traffic.
The problem is Edge Node Uplink 1 (10.10.25.101) can ping the firewall interface and vice versa, but interface 2 (10.10.26.102) cannot ping the firewall interface.
I did a packet capture on the firewall and the firewall interface 10.10.26.1 is sending ARP Broadcast when traceroute was performed from 10.10.26.102.
(https://communities.vmware.com/t5/image/serverpage/image-id/103152i557A57E5BCC97EE4/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
(https://communities.vmware.com/t5/image/serverpage/image-id/103151iB5BC8092CCC58AAC/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
I have checked the ARP table and MAC address from Uplink 1 is added but Uplink 2 there are no entries from the Edge Node. I have set up another VM (10.10.26.225) on the 10.10.26.0 network and it can reach firewall interface (10.10.26.1) without any issues, and has entries from the VM (10.10.26.225) as well.
(https://communities.vmware.com/t5/image/serverpage/image-id/103145i275C78CB74A2D1B3/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
Anyone experienced with this issue, or knows what's going on, or what can be checked ?
Sorry been on this for about a week now.
Misconfigured prefix length? /32 instead of /24 by accident?
Did you check the configuration for uplink 2? Is it on the correct VLAN, is the prefix length correct?
Your opnSense asks for the mac address and gets no answer -> I doubt this is an opnsense issue.
Both uplinks have the exact same configuration, and are in /24, and in the right VLAN as well.
Yesterday I did a firewall state table reset and the 2nd uplink started responding to ping normally, however, it still did not have any entry in the ARP Table, and upon reboot of the firewall and edge node the 2nd uplink went back to not responding to pings.
I get confused.
A state reset enables pinging and a reboot stops it. That (at least in my cases so far) means usually a misconfiguration on network side.
I am confused with two things in your original post:
You claim that
NSX-T Edge Node Uplink 1 - 10.10.25.101
NSX-T Edge Node Uplink 2 - 10.10.26.102
but in the ARP table I see an entry for 10.10.25.102 and the same mac address for two IPs of your opnsense:
10.10.25.1
10.10.26.1
So is there 10.10.25.102 in your network, too?
Hi,
Yes, there are 2 NSX-T Edges deployed, each with 2 Uplinks, and each uplink in a separate VLAN, its an Active/Active HA setup.
So :
Edge Node 1, Uplink 1 (10.10.25.101, VLAN 25), Uplink 2 (10.10.26.101, VLAN 26)
Edge Node 2, Uplink 1 (10.10.25.102, VLAN 25), Uplink 2 (10.10.26.102, VLAN 26)
Uplink 1 works with the firewall, while Uplink 2 does not.
10.10.25.1 and 10.10.26.1 are 2 VLAN interfaces that have the parent interface MAC address seen in the ARP table.
Ping from T0 VRF in Edge
edge1(tier0_sr[2])> ping 10.10.26.1
PING 10.10.26.1 (10.10.26.1): 56 data bytes
36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 0054 0000 0 0000 40 01 3230 10.10.26.101 10.10.26.1
36 bytes from 10.10.26.1: Destination Host Unreachable
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 0054 0000 0 0000 40 01 3230 10.10.26.101 10.10.26.1
Traceflow
(https://communities.vmware.com/t5/image/serverpage/image-id/103154iF54BBDF33885C797/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
(https://communities.vmware.com/t5/image/serverpage/image-id/103155i60EE68856ECD3116/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
(https://communities.vmware.com/t5/image/serverpage/image-id/103156i175F89A697FDF1BD/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
This is a test environment, using 1 firewall, and all the Edge Node uplinks are setup for BGP Peering on the same firewall.
Ok, I am afraid that does not clear up things to me. However, could you paste the actual interface configuration from OPNSense for your interfaces, esp
VLAN 25 - 10.10.25.1
VLAN 26 - 10.10.26.1
Thanks
Sorry about that.
I wasn't sure how to get interface information from CLI.
Network Overview
(https://i.ibb.co/vYWmvSy/Network.png)
(https://i.ibb.co/WnSCksD/Interfaces-Assignment.png)
(https://i.ibb.co/mCdymWL/vlan-interfaces.png)
(https://i.ibb.co/37V4LCB/int25.png)
(https://i.ibb.co/fqNc4yW/Int26.png)
I really don't see any problem here.
I guess you already checked the correct VLAN assignments on all involved (virtual and physical) network hardware. From the ARP protocol we still can tell that no ARP is returned.
Causes might be:
* Any ARP messages get filtered somewhere
* One of the ARP message directions is broken and packets are lost. (VLAN mismatch)
Given that you observed a change during state reset until the next proper reboot, I would check VLAN assignment on all involved devices and then check for any automatic or manual firewall rule on OPNsense (e.g. "block private networks" is off). The same for any possible filters involved on the ping target (if any).
Sorry for this generic advice. At least your interfaces, VLAN assignment & adressing scheme seem to be good on OPNsense.
Appreciate you taking the time, and thanks for the feed back, definitely helpful in trying to narrow down the issue.
Will definitely recheck anything associated with VLANs.
Seems like double tagging problem, 1 from vCenter Distributed Switch, and 2nd from OPNSense interface.
The issue is OPNSense VLAN interfaces cannot be created without tags, or cannot be set as 0 so tagging can be set at Distributed Switch level only.
I set the Edge Uplink portgroups to trunking.
(https://communities.vmware.com/t5/image/serverpage/image-id/103188i4AE526E16C3AF3BB/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
And firewall ARP table now has the interface attached.
(https://communities.vmware.com/t5/image/serverpage/image-id/103189iC295668ECA31BFAB/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
Now both interfaces are in Established state, and BGP peering on all Edge Interfaces successfully.
edge1> vrf 2
edge1(tier0_sr[2])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.101 Local AS: 65000
Neighbor AS State Up/DownTime BFD InMsgs OutMsgs InPfx OutPfx
10.10.25.1 65555 Estab 00:12:58 UP 46 20 12 4
10.10.26.1 65555 Estab 00:12:58 UP 46 20 12 14
Thu Aug 24 2023 UTC 17:54:55.772
edge2> vrf 1
edge2(tier0_sr[1])> get bgp neighbor summary
BFD States: NC - Not configured, DC - Disconnected
AD - Admin down, DW - Down, IN - Init, UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 10.10.25.102 Local AS: 65000
Neighbor AS State Up/DownTime BFD InMsgs OutMsgs InPfx OutPfx
10.10.25.1 65555 Estab 00:15:18 UP 48 23 12 12
10.10.26.1 65555 Estab 00:15:18 UP 51 23 12 6
Thu Aug 24 2023 UTC 17:57:02.232
Any way to have VLAN interface without tagging at firewall ?
Set VLAN untagged on the switch and omit VLAN tagging on OPNsense.
Quote from: tron80 on August 25, 2023, 08:48:03 AM
Set VLAN untagged on the switch
This is possible, vCenter Distributed Switch can be configured.
Quote from: tron80 on August 25, 2023, 08:48:03 AM
omit VLAN tagging on OPNsense.
Sorry not sure I understand how to omit tagging on OPNSense. Do you mean by creating regular interfaces ?
Just use the physical interface without the VLAN.
What exactly do you want to achieve?
Yes, exactly.
In order to access the untagged port on OPNsense you use the interface itself rather than a VLAN.
However, same question from me: Why?
Quote from: Patrick M. Hausen on August 26, 2023, 10:30:44 AM
Just use the physical interface without the VLAN.
What exactly do you want to achieve?
Thanks, this is what I plan to do now.
Quote from: tron80 on August 29, 2023, 04:14:47 PM
Yes, exactly.
In order to access the untagged port on OPNsense you use the interface itself rather than a VLAN.
However, same question from me: Why?
Thanks, I though it was possible to set up a VLAN interface without tagging.
I was trying to setup NSX Edge to send traffic over Uplinks for all VLANs over Trunk Ports (works), and for specific VLANs over Trunk ports (does not work) as I found it should work both ways, so I'm trying to understand why it does not work with the 2nd way, solely for learning purpose. I'm sure its a configuration issue, so trying to troubleshoot it but couldn't find where the issue was.
That should work, but if you expect VLANs on OPNsense to be connected in sny way across multiple ports, then that is your problem. OPNsense is not a switch. There is no "fabric". You would need to create bridge interfaces manually.
E.g. VLAN 10 on igb0, VLAN 10 on igb1, and an untagged igb2 are NOT connected in any way. As I wrote you could create a bridge with all these interfaces as members.
Quote from: Patrick M. Hausen on August 30, 2023, 10:41:26 AM
That should work, but if you expect VLANs on OPNsense to be connected in sny way across multiple ports, then that is your problem. OPNsense is not a switch. There is no "fabric". You would need to create bridge interfaces manually.
E.g. VLAN 10 on igb0, VLAN 10 on igb1, and an untagged igb2 are NOT connected in any way. As I wrote you could create a bridge with all these interfaces as members.
I understand the VLANs are not connected in anyway even though they belong to the same parent interface.
I'm just trying to understand why
when the Edge Uplink Portgroups are configured with all VLANs [0-4094], the Edge vNIC carrying VLAN 26 traffic can reach the firewall interafce but when specific VLANs are added in the portgroup [26, 24], the same Edge Uplinks that allow VLAN 26
can't reach the same firewall interface.
Could you do a quick and simple diagram of the two configurations, please?
Quote from: Patrick M. Hausen on August 30, 2023, 12:05:18 PM
Could you do a quick and simple diagram of the two configurations, please?
Network Diagram(https://i.ibb.co/MGTgj2v/Edge.png)
Distributed Switch Uplink Portgroup with All VLANs allowed (works)(https://communities.vmware.com/t5/image/serverpage/image-id/103192i38318107A63F915A/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
Distributed Switch Uplink Portgroup with only VLANs 26, and 24 allowed (not working)(https://communities.vmware.com/t5/image/serverpage/image-id/103191i4B10A248292ACCB6/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
Just adding..
As you can see the Distributed Switch Portgroup has 2 VLANs 25 and 26, only VLAN 26 faces this issue of not reaching its firewall interface, all traffic in VLAN 25 reaches its firewall interface without any issues.
Point being VLAN 25 faces no issue at all in any configuration, but VLAN 26 faces this issue, even though both uplink configurations are the same, except the VLAN ID.
Sorry - no idea. Use tcpdump or in case of real switches a monitor port to watch the packets. That's what I would do.
Does the source send the packets down the right path? Are they correctly tagged/untagged?
Does the next system in line receive the packets?
Repeat for each next hop.
Thanks,
I have captured packets on the firewall when VLAN 26 cannot ping its firewall interface.
(https://communities.vmware.com/t5/image/serverpage/image-id/103152i557A57E5BCC97EE4/is-moderation-mode/true/image-dimensions/2500?v=v2&px=-1)
The firewall parent interface sends an ARP Broadcast.
Appreciate all the help, will go through, and try and rebuild again, must be something I missed.
Thanks again..