I have a range of hosts that are unable to resolve or route to WAN hostnames.

[ayylmao]

platform: Nutanix

So I'm trying to segment my network a bit to compartmentalize and even clawed back my config to make this as simple as possible to get things up. But the snag I'm running into is I can't seem to resolve hostnames from the Nutanix AHV or CVM nodes which is preventing me from registering my CE cluster with their NEXT servers.

I've kind of been banging my head against the wall creating, deleting, and modifying firewall rules without any luck.

I have your usual 192.168.1.0 "user" network, making my primary gateway to the WAN 192.168.1.1 on the LAN interface.

192.168.1.2 is assigned to a VE (virtual ethernet) port on a switch (ICX7250) and a specific amount of 1 GbE ports are also assigned to this VLAN. Then, on that same switch the nodes are connected via 10 GbE only in their own VLAN/network, specifically 10.0.6.0. This second "VLAN" has a VE port assigned 10.0.6.1. So, the nodes/CVM all have 10.0.6.1 as the gateway. This switch has only a single default gateway setup back to 0.0.0.0/0 with 192.168.1.1 as the next hop.

In OPNSense I have a gateway defined on the LAN interface with address 192.168.1.2 so that I can setup a route to 10.0.6.0/24 via 192.168.1.2.

Everything is pingable between each other, private network wise. But the issue is hosts 10.0.6.2 through 10.0.6.10 cannot ping any WAN based addresses. Literally every other single IP on the network can do successful pings against the basics, google.com, nsc01.nutanix.net, etc.

OPNSense runs Unbound DNS by default and I'm not sure if this is causing issues. So far I've deleted all rules and forwards that I don't think are doing anything or helping and I've just got a single rule on the LAN interface with 10.0.6.0/24 as a source and a destination of 192.168.1.1/24 to allow. 192.168.1.1 is pingable from all AHV/CVM nodes so I'm not really getting what the issue is.

When I do a tracepath nsc01.nutanix.net the initial hops are to 1? LOCALHOST, 1: gateway, another 1: gateway and then subsequent no replies.

I'm really rudimentary with networking so hoping someone has an idea of what's needed here. I'll admit I'm a little confused with how the whole interface-based firewall roules tends to work in OPNsense. I'll be cross posting this in other communities to get more eyes on it.

edit: for what it's worth I've been using the firewall live view to try and get a sense for where things are either being blocked or passed and created many rules to turn reds to greens, but they don't seem to matter ultimately. I can successfully get "passed" traffic out from say pings, but these hosts I mentioned still aren't able to perform lookups because the ping command times out. So I am not sure if I need some kind of bi-directional rule?

[ayylmao]

I simplified my setup even further by simply plugging in a physical link between a 10 GbE interface on my opnsense device, assigning it an IP in the 10.0.6.0 subnet (let's say .11 since I didn't want to go back and re-do everything in Nutanix) and then assigning the port on the switch side as an untagged member of that "VLAN". After doing this I am able to authenticate my cluster/reach the internet from these devices.

So I wonder if it has something to do with how I had my VLANs setup between 192.168.1.0 and 10.0.6.0.

The way it looks is I have ports 1 through 24 (1 GbE) in the default VLAN 1 untagged, and then ports 25 through 28 (10 GbE) in VLAN 3 untagged. VLAN 1 has a virtual interface of 192.168.1.2, but the physical port that is linking this "VLAN" back to the opnsense interface 192.168.1.1 is also untagged. It hasn't been made clear to me if this singular physical port needs to be tagged.

I have not configured VLAN at all on the opnsense side.