Problem with tagged VLANs - works only with tcpdump running...

[jewe57]

Hi,

I am new to OPNsense. I work with Cisco ASA and CheckPoint-Firewalls since over 20 years and I wanted to test OPNsense for my privat use and to have a payable alternative for my smaller customers.

So I understand the structure of firewalls, but OPNsense seems to be a bit different - and I am running into a problem since days.

My Gateway-Machine at home has 2 NICs and both are running fine without VLAN-Tagging. With tagged VLANs on I have following problem and did not find anybody with the same experience.
The machines in the VLANs are getting their DHCP-addresses from the OPNsense - so I am pretty sure the tagging itself is working. But after that, no traffic will pass the gateway. When I switch one of the VLANs from tagged to untagged (configure new interface in OPNsense), the traffic passes - with the same rules, nat and dhcp-config.

Where can I start my debugging?

OPNsense 24.1.3_1-amd64
FreeBSD 13.2-RELEASE-p10
OpenSSL 3.0.13

[Seimus]

So you say,

A. Hosts in VLAN are able to obtain DHCP from OPNsense
B. Hosts are not able to communicate afterwards thru OPNsense
C. No traffic will pass the gateway

May I ask
What do you mean by "No traffic will pass the gateway"?
You dont see any traffic on ingress? Or egress?
GWs are created on OPNsense?

Can you enable logging on all of the rules and check in Live view, or run capture from OPNsense on Ingress Egress and see if the packet came to and thru OPN?

Also do you run Baremetal or VM (Proxmox, etc.)?
Can you ping from hosts in a specific VLAN the OPN at all?
Can you trace from hosts in a specific VLAN to OPN?

Regards,
S.

[jewe57]

1. it's a baremetal box
2. hosts in the vlan can ping inside the vlan (switch has IP, replies), but they can't ping the firewall, can't connect to the firewall and can't communicate through the firewall (neither ping nor higher services).
3. in the live view of the log nothing is shown, looks like no traffic coming in.

Traceroute and tcpdump will be my next step in a few hours...

[Patrick M. Hausen]

2. hosts in the vlan can ping inside the vlan (switch has IP, replies), but they can't ping the firewall, can't connect to the firewall and can't communicate through the firewall (neither ping nor higher services).

Then please show the firewall rules for that VLAN interface.

[jewe57]

I am not at the PC atm, back in a few hours. But I did a ,,permit any any" for the test.

[jewe57]

Okay, I have a maximum of 4 attachments per post, so I have to split into 2 messages.

Attached are screenshots of my floating rules and the rule of my IOT-Interface (every tagged VLAN lookes more or less the same).
Also attached is the vlan-configuration and the assignments-page.
More will follow in the next post.

[jewe57]

Next post.

BUT FIRST:
I started a tcpdump on the OPN for interface vlan0.30 (just tcpdump -i vlan0.30, no filter) - and the IOT-Machine got a reply to the ping from my gateway!!! After stopping tcpdump the ping failed. Starting tcpdump again, ping was replied - stopping tcpdump also stops the ping-replies....
Same for higher traffic like http...
traceroute 8.8.8.8 on the IOT-Device shows 1 * * * (and so on) - with tcpdump on the gateway it shows 1 10.0.0.1 2 192.168.1.1 - and then internet...

WTF ????




Attached is a screenshot of the IOT-Interface-Config (Vlan 30) and here you'll see the
ifconfig (shortened).


My WAN-Interface is behind my internet-router, so it has an RFC1918-Address...

>ifconfig

igc0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN (lan)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether f4:c8:8a:9c:42:01
        hwaddr 00:e0:4c:6f:0f:8f
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN (wan)
        options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,NOMAP>
        ether 00:e0:4c:6f:0f:90
        inet 192.168.0.100 netmask 0xfffffc00 broadcast 192.168.3.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
(...)
vlan0.30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: IOT (opt3)
        options=4000000<NOMAP>
        ether f4:c8:8a:9c:42:30
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
        groups: vlan
        vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
(...)
vlan01: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Guest (opt1)
        options=4000000<NOMAP>
        ether f4:c8:8a:9c:42:01
        inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
        groups: vlan
        vlan: 11 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

[zan]

You need to enable promiscuous mode on your igc0 interface because one of your vlan interfaces has a different MAC address than parent interface's, otherwise it won't be able to see vlan traffic.
Thats why it works when tcpdump is running because tcpdump turns on promiscious mode on interface it is tracking.

[jewe57]

oh, that was too easy. Shame on me ;-)

Coming from other firewalls like Cisco or CheckPoint this worked automatically, so I learned alot today :-)

Thank you - problem solved!