VLAN mysteries

[toxic]

Hello.

I'm having difficulties finding out why my VLAN is not working...

My setup has 2 hosts on a TL-SG108PE switch that has VLAN 802.1Q enabled and configured for them to talk on VLAN 100, and my opnSense box is also plugged into the switch and will get VLAN100 traffic (tagged).

Both hosts do manage to get a proper IP by using DHCP on VLAN 100 : 10.0.100.x
Both hosts can ping eachother.

But, none of these hosts can ping the opnSense box (despite the fact that it did provide them with an IP, I have no other DHCP running...)

Also, the opnSense box cannot ping either of these hosts.
I have even checked, arp 10.0.0.100 running on the opnSense box does return the proper MAC for the host on VLAN 100, it says interface bridge1 as should be... still, ping always fails...

On the opnSense Box, I have quite the setup :

• igb0 and igb1 are in a static link aggregation with the Tp-Link switch : lagg0
• igb2 and igb3 are in a LACP link aggregation with my Synology NAS : lagg1
• em0 is a virtual interface provided by proxmox, the host OS running openSense in a VM and passing all 6 NIC (igb0-5) with pci-passthrough. This em0 is used to talk to the proxmox host when I'm on the network routed by the VM. It's setup fine, working.
• bridge0 contains lagg0+lagg1+em0, no particular VLAN tag (default 1 on my switch), static IP 10.0.0.1, all clients running properly, as soon as they are on VLAN1 they can ping the opnSense box and reverse is true, they get internet, all fine.

I then created a VLAN tagged 100 for each of the interface in my bridge0 (lagg0+lagg1+em0) beacause creating a VLAN on the bridge0 interface seemed not to work (ifconfig showed vlan to stay at 0.. no DHCP was working...)
I then created the following assignations :

• LAGG0_100 : vlan 100 on lagg0 (that's how it show up in the dropdown of interface assignment page...)
• LAGG1_100 : vlan 100 on lagg1
• EM0_100 : vlan 100 on em0
• bridge1 : LAGG0_100 + LAGG1_100 + EM0_100

I created a LAN100 interface assigned to bridge1, set it to static IP 10.0.100.1 and enabled DHCP for LAN100, and this is the current state where DHCP assigns properly in the 10.0.100.100-200 range, ping works between VLAN100 members (I assume it never leaves the single switch I have... doing packet capture on LAN100 interface seems to confirm this) but ping from a VLAN100 member with the opnSense box fails in both directions... (packet capture on LAN100 does show me an arp request for 10.0.100.1 which is the opnsense IP on VLAN100, with the proper response with the MAC of LAN100, but then no ping coming in and no response...)

All interfaces are set to enables but only the bridge0 and bridge1 are set to a static IP.

I'm only using IPv4.

Just for the lolz, I removed the LAGG0_100 interface from bridge1, and reassigned the LAN100 which has the static IP and DHCP server, and gave it the "vlan 100 on lagg0" interface, and for some miracle reason, the opnSense box can now ping both hosts with their VLAN100 IP and reverse also true.
One of my host is actually an openWRT access point that is setup to use both VLAN 1 and VLAN 100, it can still ping an be pinged from everyone.
But of course, my Synology NAS is now fully outside of VLAN 100 since lagg1 is still in the bridge1 but the bridge1 itself is not used...


I thouht that some tunablesmight be the cause of the issue. Here is what I have now :
    net.link.bridge.pfil_onlyip=default (0)    
    net.link.bridge.pfil_local_phys=default (0)    
    net.link.bridge.pfil_member=0    
    net.link.bridge.pfil_bridge=1
but leaving them all as default or as is described above seems to make no difference...

I added a floating rule for the interface to allow from any to any on all... still, no ping.
I tried also after I did pfctl -d and it did not change anything...

Is there any hope for me to get the bridge1 back working so I have VLAN100 across all ?

Any hint as to why the behaviour of DHCP working and ping not is welcome, as well as any help in setting up VLAN 100 working on bridge1...

I'm going to try again with these tunables but the VLAN created on bridge0 instead, we'll see...

[banym]

Just to  be sure. Do you allow ICMP echo request on that interfaces?
In other words, have you verified that ping does work for all devices you debugg with?
If ARP and DHCP work do you see the leases in your DHCP service?

[toxic]

Hello,

Thanks for your reply and interest.

I did add allow ICMP in the FW rule, nothing specific added for DHCP but I think it gets autogenerated when enabling the DHCP.

I do see the leases in the DHCP service indeed.

For further mysteries, in the last option where I put the VLAN on the LAGG and not on the bridge itself (preventing me from adding it to a bridge later), I finally managed to have the ping working between VLAN memebers and the opnSense box.
But when I tried to allow trafic from VLAN 9 (10.0.9.x) to VLAN 100 (10.0.100.x), I had to add a gateway manually for the VLAN 100 interface on the LAGG, set the IP of this GW to the veryIP opnSense is designed to use itself statically on this VLAN (10.0.100.1), and I found that an ICMP allow rule that overrides the gateway from default to this newly created gateway was allowing me to get the ping working between VLANS even without disabling all other rules (ip 10.0.9.2 pinged 10.0.100.2 for example). But the same rule to allow TCP traffic overt port 443 did not work at all...

Getting tired with VLANs that don't behave like I thought they would... Maybe I need to disable the bridge0 that is not tagged as part of a VLAN ? Would keeping it pose an issue ? I thought it should not and would only handle untagged traffic...