Gateway not accessible from particular VLAN, but some devices can access WAN

[waldorf]

Using v24.7.1

I have one VLAN (also has MDNS repeater configured for this interface) from which I am unable to ping the gateway. (192.168.3.1)

When connected to this VLAN through WIFI AP (seems same when using wired) :

- I get a DHCP IP address with the correct gateway IP
- When doing a network scan, I see connected devices on the same VLAN
- Able to ping those devices (e.g. Nvidia shield 192.168.3.60)
- Unable to ping the gateway IP
- Unable to ping an IP on the internet (which makes sense if the gateway is not reachable
- And now the most weird, the Shield on 192.168.3.60 is able to access the internet (Netflix playback works)
- I just created an access port on the (L2) switch which is directly connected to the opnSense box (this to exclude other switches which might be misbehaving), behavior is the same. I am able to ping devices in the 3.x network, but unable to ping the 3.1 gateway. (I can ping the 3.1 gateway from another VLAN)
 
There are some firewall rules, but currently the first (enabled) rule on this interface is an ipv4* allow any to all.

At the moment I have no clue what this can be, any suggestions ?

[viragomann]

Is there an outbound NAT rule in place for this VLAN?

[waldorf]

The only outbound NAT rules are the default ones. I never added/changed any NATY rules.
I seen 2 auto generated rules listed, which both contain all network names for source network.
I just noted I have a 2nd VLAN which also cannot ping it's own gateway (but has DHCP  connected clients)

[viragomann]

So OPNsense has automatically added an outbound NAT rule for this VLAN.

On OPNsense apart from this, only a firewall pass rule to any is needed for internet access, and you said, you've created one.
So possibly the issue is outside of it, on the switch?

On OPNsense you can run packet capture on the internal VLAN interface and on WAN to investigate the traffic flow.

[waldorf]

One of the switched was something I suspected as well. That was the reason I created an access port for this VLAN  on the switch directly connected to OPNsense.

I have to say, this is a switch which I bought recently, so there might be something I overlooked.
(HP Aruba 1830)
 
I have a simple VLAN capable switch as spare and will connect it to the OPN sense trunk port to see what this does (when only using the particular VLAN) 

[waldorf]

Just tried the following using very simple switch with tagged VLAN support.

opnSense trunk port (Only tagged) -> switch port 1 tagged VID 13 -> port 2 untagged VID13 -> laptop

Laptop receives DHCP IP address 192.168.3.x but ping 192.168.3.1 does not work.
Neither does ping 8.8.8.8 or 1.1.1.1 work.

So it really seems something with opnSense config. In the past the setup worked, so something must have been changed, although I do not know what. 

It seems I must dig into the config files.

[Patrick M. Hausen]

If you just added that tagged VLAN on OPNsense you also need to create a firewall rule to actually allow any traffic from clients in that interface.

DHCP is passed by automatic rules, that's why address acquisition works.

[waldorf]

I didn't change the OPNsense config, just pulled out the trunk wire from the main switch and put it in another switch (different brand/type), just to exclude switch issues.  This temporary switch was configured to have a single tagged VLAN 13 port and an access port with the same VLAN untagged.  Behavior the same, client receives IP address but gateway not accessible.

I have 2 other VLANS which just work as expected, Only VLAN 13 not. (The first firewall rule it currently has is an "allow all IPV4* to any"  for this interface.

One thing I just noticed: the 'interfaces menu'  has a 'neighbors' submenu. This neighbors menu shows several entries (which I never added) and all of those entries are from the  non working VLAN network. Can this be related somehow ? And when are entries automatically added to the neighbors list ?

I enabled logging for the "allow" any rule for the particular VLAN, did a ping to the gateway from a client. All the ICMP requests were green, but ping did not get a response.   

Same client on another VLAN no problem.

It somehow seems the allow-all in-rule is not working. But I have no idea why. I do not use any floating rules.

And strangely the Shield on the same VLAN is able to play Netfix.

[waldorf]

To answer my own question. It turned out it was caused by the "enable static ARP" which was enabled and which does a little bit more than only mapping IP addresses to MAC addresses.