Traffic on same interface blocked?

[Patrick M. Hausen]

Hi all,

so I have this TrueNAS SCALE system with "apps" running in a Kubernetes/Helm managed environment. The NAS and the apps are connected to vlan02 with IP address range 192.168.2.0/24.

Since that NAS has got more interfaces I need to explicitly tell Kubernetes which interface to use for egress. The settings (on TrueNAS) are:

Node IP: 192.168.2.11
Route v4 Interface: vlan2
Route v4 Gateway: 192.168.2.1

So far so good. 192.168.2.1 is OPNsense and all apps can reach the Internet and other services just fine.

Now I installed an app named Uptime Kuma which is a fancy Nagios/Icinga replacement for this century. Really nice. And it also works well, but ...

When I try to monitor systems that are in the very same broadcast domain - 192.168.2.0/24, vlan02 on my OPNsense, this Kubernetes thingy decides not to use ARP but throw the packets at my OPNsense (192.168.2.1) instead.

This triggers the "Default deny / state violation rule".

When I add a rule on that interface:

Source: TrueNAS host, i.e. 192.168.2.11
Destination: SRV net, i.e. 192.168.2.0/24
Protocol: TCP
Destination Ports: 80, 443
Action: allow

then the monitoring system works as it should. OK, (ab)using the default gateway instead of using ARP is not what the system is supposed to do, so that is also expected and very much consistent.

The question is:

Why doesn't Firewall > Settings > Advanced > Static route filtering - Bypass firewall rules for traffic on the same interface achieve the same effect?

Isn't that setting supposed to turn OPNsense into a router for all traffic that goes from and to the same interface? (and send an ICMP redirect, of course ...)

Thanks and kind regards,
Patrick

EDIT: added a screenshot of the rule that makes it work. Host4_TrueNAS is 192.168.2.11 and very well in the SRV net (192.168.2.0/24) network.

[Patrick M. Hausen]

So the root cause seems to be a bug in the TrueNAS networking implementation. I can live with that. To get to a clean network setup and a manageable set of rules I simply moved the k3s/apps interface of my TrueNAS into its own VLAN, so of course all apps throwing every packet at the default gateway is ok, because there are no other services in that local broadcast domain.

Still I would really like to know why

Firewall > Settings > Advanced > Static route filtering - Bypass firewall rules for traffic on the same interface

isn't doing what I think it should be doing. I know the issue from decades of working with firewalls and e.g. the Sidewinder firewall had a setting for that. If source and destination are on the same interface just route, don't apply any firewall rules.

Is that the intention of this OPNsense setting? If yes, why doesn't it work as intended? I would be willing to contribute some effort into debugging and getting this feature functional. If no, what is the intention of that setting? Could the wording in the UI be improved, possibly?

Kind regards,
Patrick

[LOTRouter]

Personally, I'd rather that OPNsense respond exactly as it is described.  I'd hate to try troubleshooting why performance to my NAS on the same LAN/subnet was poor, only to discover that all traffic had to be routed through my firewall, when it obviously shouldn't be.  I don't see creating a rule to allow it to route within the same subnet as a viable solution to allow my NAS to beat the crap out of my router.  At least now you know the root issue is with your NAS and you can allow it if you choose, but I wouldn't.

[Patrick M. Hausen]

It's not the NAS function that is behaving badly, only the "apps" with their weird kubernetes based networking.