Switch with port isolation + OPNsense: Allow communication between devices

[Apollo3zehn]

Hi,

I am setting up a network with a switch and a firewall (OPNsense). Devices connected on the switch should generally not be able to communicate to each other except in rare cases. Internet access is provided by the firewall.

I thought it is a good idea to configure port isolation (private VLAN) on the switch so that port-to-port communication on the switch is forbidden and all communication must go through the firewall*. I would then, in theory, be able to configure a simple firewall rule to allow specific traffic, e.g. device 1 can access device 2 via SSH.

But that is not working I don't know if it is a bug, wrong configuration or not possible by design. The problem is that packets that go from the switch to the firewall are not sent back and so there is no communication between both connected devices.

My question is: Is it somehow possible to sent packets back to the network interface it entered originally?

In case it matters: I have grouped most of the firewall's network interfaces into a transparent filtering bridge. With tcpdump I can see that packets entering on one interface are forwarded to all other interfaces except the one it entered originally.

I hope you can point me in the right direction (or tell me that this is not possible). Thanks in advance!

* I know that I can configure the switch to use "community ports" to allow specific devices to communicate to each other, but I would prefer to use the firewall as it allows much more fine-grained control over what is allowed and what not. Community ports would allow everything (all ports, all protocols) which is not desired.

[Patrick M. Hausen]

A layer 3 firewall cannot filter traffic between hosts on the same network / prefix / broadcast domain (3 terms for the same thing). The hosts will try to communicate directly because source and destination are in the same network. They try and fail because of the switch. End of story. The never try to use the default gateway.

[Apollo3zehn]

The hosts will try to communicate directly because source and destination are in the same network. They try and fail because of the switch

The packets from the first device connected to the switch reach the firewall because the port the firewall is connected to the switch is configured as a "promiscuous" port. This is the port where all packets leave the switch, no matter where they are addressed to. So yes, the devices don't use the firewall as gateway but the packets still reach and enter it. Firewall rules are also applied (I can see it all via tcpdump) but it doesn't matter because the packets are not sent back to the switch and never reach the second device :-(

[Patrick M. Hausen]

Are the packets directed at the MAC address of OPNsense on layer 2? I would expect the sending host to send ARP requests for the destination, which will fail because of the isolation. Does the switch perform some magic here?

If the destination MAC address is not the one of OPNsense, the firewall won't act upon it.

[Apollo3zehn]

Are the packets directed at the MAC address of OPNsense on layer 2?

No, the destination MAC address is that of device 2. Because of the port isolation the packet is not delivered directly but leaves the switch on the promiscuous port, where the firewall is connected to. You are right that in that case the firewall would normally ignore that packet but I configured the firewall as a transparent filtering bridge (according to this: https://docs.opnsense.org/manual/how-tos/transparent_bridge.html) which means it will forward (and filter) all incoming packets to all interfaces that are part of that bridge. Except the interface the packet entered originally, which is the problem I have. I would like the firewall to ping the packets back to the entering interface to reach the second device.

[Patrick M. Hausen]

The transparent filtering bridge is not a learning bridge in the classical sense. It's really more like an Ethernet cable with some sort of magic filter in the middle.

Can you connect a second port of OPNsense to the switch for egress and have the switch deliver it to the destination? Looks like OPNsense needs a hand here in proper forwarding 

[Apollo3zehn]

I will try this tomorrow or next week. I have to find out if DELL supports more than one promiscuous port per private VLAN. Thanks so far :-)

[dseven]

Connecting two bridged ports to the same switch seems like a good way to create a loop. Maybe you can kludge around that with spanning-tree, or/and hope that the "private VLAN" functionality on the switch breaks the loop.... but the whole idea seems rather hokey to me. If you really need fine-grained control at layer 2, maybe get a switch that supports ACLs?

[Apollo3zehn]

Yeah I though the same yesterday about what will happen with two promiscuous ports. Will both get all packets, or only one of them, and if so, which one? And so on. I have a switch which seem to support ACLs but I was not aware of it until now. Thanks for that hint, I think this will be the solution.

[Apollo3zehn]

It seems to work with ACLs :-) Thanks again