[SOLVED] Unable to prevent traffic from VLANs to untagged LAN

[sandman0815]

Hi all,
I'm using OPNsense now for about a year in my Homelab and I'm totally happy with it. Up to now I have been able to solve any questions or problems that I had with the help of the documentation, tutorials and posts in this forums.
Unfortunately I'm a bit stuck right now...

I'm running OPNsense as a VM on Proxmox. I have passed through the NICs for WAN and LAN connection and there is a third connection which is attached to a bridge on the proxmox host which bridges the connection to the LAN port of my modem (separate IP subnet - this is only for testing purposes). On OPNsense's LAN interface there are some VLANs as children and the untagged traffic should be my Management LAN (Switch is configured accordingly).
The proxmox host itself does have a further bridge, which is more or less the Proxmox default bridge VMBR0, but with VLAN awareness. It is connected to the hardware switch without any further VLAN configuration to just reside in my Management LAN and is used to access the Proxmox WebUI.

I'm using VLAN 10 as my "internal" Network with my regular devices and VLAN 50 for the DMZ.

The problem:
With devices in any VLAN I'm able to reach (ping, cURL etc) devices in the untagged (aka Management) network area. How can I find out, why this is possible and how can I prevent that? I can't find any firewall rule that makes this possible.
Btw. it is not possible e.g. for devices in DMZ to reach devices in INTERNAL, so inter VLAN routing seems to be blocked as expected.

Trying to understand what is going on, I made a packet capture on the LAN interface (in promiscuous mode) and made the following observation:
Running a ping from an iPhone in the INTERNAL (=VLAN 10) network to the Proxmox host on the Management LAN.
First occurance of the ICMP packet does have the correct VLAN tag and is sent from the iPhone devices MAC to the OPNsense LAN interface MAC:
VLAN TAG: 10
SRC 28:34:ff:d7:3e:b6 (Client iPhone, ip=10.99.110.21)
DST a8:b8:e0:02:4d:5f (OPNsense igc1 = LAN Interface, ip=10.99.1.1)

Then, the VLAN tag seems to get stripped and the packet is forwarded to the Proxmox host:
VLAN Tag: none
SRC a8:b8:e0:02:4d:5f (OPNsense igc1 = LAN Interface, ip=10.99.1.1)
DST a8:b8:e0:02:4d:71 (pmxhost enp7s0 = vmbr0, ip=10.99.1.50)

Proxmox Host responds and packet goes back to OPNsense LAN interface:
VLAN Tag: none
SRC a8:b8:e0:02:4d:71 (pmxhost enp7s0 = vmbr0, ip=10.99.1.50)
DST a8:b8:e0:02:4d:5f (OPNsense igc1 = LAN Interface, ip=10.99.1.1)

VLAN Tag seems to be added and answer is sent to iPhone:
VLAN Tag: 10
SRC a8:b8:e0:02:4d:5f (OPNsense igc1 = LAN Interface, ip=10.99.1.1)
SRC 28:34:ff:d7:3e:b6 (Client iPhone, , ip=10.99.110.21)

Can anybody help me to understand what's going on there?

Warm regards,
sandman

[dseven]

Oh my! The VLAN police are going to lock you up for mixing tagged and untagged on the same physical link! ;D That aside...

I assume that the IP addresses noted in your packet capture description are those associated with the MAC addresses, and not the actual addresses from the IP headers in the captured packets?

Without seeing your firewall rules, it's hard to say... but it does sound like there is one that is allowing the traffic. What rules do you have on your VLAN 10 interface?

[sandman0815]

Thanks for you reply!

Oh my! The VLAN police are going to lock you up for mixing tagged and untagged on the same physical link! ;D That aside...

Well, I'm not aware that this may be causing issues, but I'm doing all this to learn... So, if there is reason not to do so, I might consider changing that in my setup!

I assume that the IP addresses noted in your packet capture description are those associated with the MAC addresses, and not the actual addresses from the IP headers in the captured packets?

Yes, these are the IP addresses associated the respective MAC addresses as they would show up in the ARP table.

Without seeing your firewall rules, it's hard to say... but it does sound like there is one that is allowing the traffic. What rules do you have on your VLAN 10 interface?

Sure! These are the rules, that I created on my own:


These are some floating rules, I have created:


And finally the automatically generated rules:


Thanks again to anybody trying to help. I'm out of ideas   :'(

[dseven]

So I assume that your management network is supposed to be covered by your PrivateNetworks alias? Double-check that?

I'd maybe try adding an explicit block rule at the top of the list for INTERNAL and see if that gets applied, then try to figure out why some other rule is allowing more than you want. You could turn on logging for your (suspect) rules and try to use that to find out which one is doing it too.....

Regarding mixing untagged and tagged; there are some that say it's a terrible thing to do and will cause all sorts of problems. I've been doing it myself for some time, and not having any problems, and there's no fundamental reason for it to not work....

[sandman0815]

Thanks, much appreciated!
I'll follow through that tomorrow and will let you know...

[sandman0815]

So I assume that your management network is supposed to be covered by your PrivateNetworks alias? Double-check that?

Okay, my alias should be fine:

I'd maybe try adding an explicit block rule at the top of the list for INTERNAL and see if that gets applied, then try to figure out why some other rule is allowing more than you want. You could turn on logging for your (suspect) rules and try to use that to find out which one is doing it too.....

Thank you for this valuable hint. It has pushed me in the right direction. After adding a "block everything from anywhere to anywhere" rule at the INTERNAL VLAN, I was still able to ping hosts on the management LAN. I turned on logging for all floating rules (thanks for mentioning the logging!) and discovered a rule that was not configured correctly.
I have a rule that allows traffic from different VLANs to the AD server, but I have specified a wrong subnet mask (and obviously I have been very lazy while specifying it):


With that fixed, everything now seems to work as expected...

Thanks again for pointing me in the right direction!