Interface Routing, NAT, Firewall (arp table incomplete)

[suntzu2100]

Interface Routing, NAT, Firewall (arp table incomplete)

OPNsense 22.7.10_2-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022

Hardware: Protectli FW4B
(Gateway, Firewall, Router, VLAN)

--------------------------------------
Interfaces
--------------------------------------
WAN
LAN
NSX (Vmware nsx-t)      > 172.16.30.1
VMO (Vmware vmotion) > 172.16.20.1
VLAN30 NSX-T
VLAN100 NSX-T TEP

--------------------------------------
Firewall
--------------------------------------
WAN > Inside Any to Outside Any > Inbound 1194 UDP > Any
LAN > Any > Any > Any
NSX > Any > Any > Any
VMO > Any > Any > Any
VLAN30  > Any > Any > Any
VLAN100 > Any > Any > Any

=====================

Opnsense NSX Port: 172.16.30.1
<------------------>
Switch NSX Ports 1 to 8:
Port 1: 172.16.30.2
Port 2: 172.16.30.31
Port 3: 172.16.30.32
Port 4: 172.16.30.33
Port 5: 172.16.30.34
Port 6: (empty)
Port 7: (empty)
Port 8: (empty)
======================

[root@esxi1:~] ping 172.16.30.2
PING 172.16.30.2 (172.16.30.2): 56 data bytes
64 bytes from 172.16.30.2: icmp_seq=0 ttl=64 time=1.830 ms
64 bytes from 172.16.30.2: icmp_seq=1 ttl=64 time=2.243 ms
64 bytes from 172.16.30.2: icmp_seq=2 ttl=64 time=2.187 ms

--- 172.16.30.2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.830/2.087/2.243 ms

The switch can be pinged from the esxi host

======================
* The hosts 31, 32, 33, and 34 arp table completes and shows each host (but .1 incomplete)

Example host arp table:

[root@esxi1:~] esxcli network ip neighbor list
Neighbor       Mac Address        Vmknic    Expiry  State  Type
-------------  -----------------  ------  --------  -----  ----
192.168.1.201  00:0c:29:b7:88:1e  vmk0    1196 sec         Dynamic
192.168.1.200  00:50:56:b2:7a:89  vmk0     817 sec         Dynamic
192.168.1.1    00:e0:97:1b:b1:05  vmk0     639 sec         Dynamic
192.168.1.2    00:11:32:f6:8e:0e  vmk0     829 sec         Dynamic
192.168.1.115  00:0c:29:06:a7:d8  vmk0    1198 sec         Dynamic
172.16.40.2    00:11:32:f6:8e:0d  vmk2     731 sec         Dynamic
172.16.40.44   00:50:56:60:3a:ef  vmk2    1016 sec         Dynamic
172.16.40.42   00:50:56:63:31:37  vmk2     964 sec         Dynamic
172.16.40.43   00:50:56:65:c8:69  vmk2    1040 sec         Dynamic
172.16.30.2    c0:06:c3:f2:4e:bb  vmk3    1176 sec         Dynamic
172.16.30.34   00:50:56:60:b6:37  vmk3    1172 sec         Dynamic
172.16.30.3    00:e0:97:1b:b1:07  vmk3    1198 sec         Dynamic
172.16.30.32   00:50:56:64:3b:6a  vmk3    1164 sec         Dynamic
172.16.30.1    (incomplete)       vmk3      -4 sec         Invalid
172.16.30.33   00:50:56:69:00:e2  vmk3    1168 sec         Dynamic
172.16.30.4    (incomplete)       vmk3     -10 sec         Invalid

---------------------------------------
-------- The Problem ----------------
---------------------------------------
An esxi host (172.16.30.31) is connected to a physical NIC (vusb2) via vmk3 with the NIC connected to dedicated NSX switch [TL-SG108E] with four esxi hosts on the network 172.16.30.0/24 and all hosts can ping (ICMP) between hosts.

When I ssh into the opnsense shell and use tcpdump via the shell I can see:

Code Select Expand

08:51:23.883153 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.16.30.1 tell 172.16.30.31, length 46
08:51:23.883170 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.16.30.1 is-at 00:e0:97:1b:b1:07 (oui Unknown), length 28
08:51:24.825698 c0:06:c3:f2:4e:bb (oui Unknown) > Broadcast, RRCP-0x25 query
08:51:24.883642 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.16.30.1 tell 172.16.30.31, length 46
08:51:24.883663 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.16.30.1 is-at 00:e0:97:1b:b1:07 (oui Unknown), length 28
08:51:25.826930 c0:06:c3:f2:4e:bb (oui Unknown) > Broadcast, RRCP-0x25 query
08:51:25.884503 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.16.30.1 tell 172.16.30.31, length 46
08:51:25.884524 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.16.30.1 is-at 00:e0:97:1b:b1:07 (oui Unknown), length 28
08:51:26.830300 c0:06:c3:f2:4e:bb (oui Unknown) > Broadcast, RRCP-0x25 query

*These seems to be no communication "back" from opnsense to the hosts?

Any ideas? Within opnsense are static routes best practice for this type of topology?