Hi,
I have 5 cheap L2 Trendnet switches (https://www.trendnet.com/products/edgesmart-switch/8-port-gigabit-EdgeSmart-switch-TEG-S80ES (https://www.trendnet.com/products/edgesmart-switch/8-port-gigabit-EdgeSmart-switch-TEG-S80ES)) that I would like to deploy on home network behind a Qotom device (6 ports) running OPNSense. First I tried the topology shown in diagram-01 below but only switch-A worked in this case. The VLANs on switch-B could neither access VLANs on switch-A nor OPNSense.
As an alternate, I'm thinking of combining the unused ports on OPNSense device to create a bridge and create VLANs on the bridge and connect each port of OPNSense to corresponding VLAN port on switch-A. And use port-1 on both switches to connect each other.
I'd need to take down my network to even try this one, so wanted to check in this forum if anyone has faced the same problem as I did in diagram-1 and if the alternate topology (diagram-2) will work in theory.
Thanks.
Don't use the bridge, you'll regret it.
Post pictures of your switch vlan config.
You can use a bridge on top of a VLAN, e.g. to bridge a VLAN interface and a physical port. You cannot use VLANs over a bridge interface.
Are there any updates on making a vlan on the bridge?
Currently I have setup a bridge with 2 vlan interfaces that I want to allow to freely communicate with each other on the same vlan, and allow routing to external network via the opnsense. To get this working I have one vlan interface with ip while the other is empty.
Problem is that it often gets connectivity issues to the opnsense ip and back, while it can (mostly) communicate well between the vlan interfaces.
I would like to keep some form of gating between the two interfaces so that I do not allow all vlans to pass through the other switch, so simply connecting the two switches is not a viable solution here. Ideally I would want that the traffic on this bridge to be untagged when it goes from vlan port to the bridge and re-tagged when it goes through the vlan.
Quote from: lecris on August 18, 2022, 09:58:38 AM
Are there any updates on making a vlan on the bridge?
This is fundamentally impossible in the FreeBSD network architecture and not going to change. The bridge interface does not support tagged frames.
Quote from: lecris on August 18, 2022, 09:58:38 AM
Currently I have setup a bridge with 2 vlan interfaces that I want to allow to freely communicate with each other on the same vlan, and allow routing to external network via the opnsense. To get this working I have one vlan interface with ip while the other is empty.
Problem is that it often gets connectivity issues to the opnsense ip and back, while it can (mostly) communicate well between the vlan interfaces.
You need to assign the IP address to the bridge interface and leave both VLAN interfaces without an IP address.
This is also a fundamental property of FreeBSD and documented in the FreeBSD handbook. A bridge
member interface MUST NOT have an IP address.
HTH,
Patrick
Quote from: pmhausen on August 18, 2022, 10:34:25 AM
You need to assign the IP address to the bridge interface and leave both VLAN interfaces without an IP address.
This is also a fundamental property of FreeBSD and documented in the FreeBSD handbook. A bridge member interface MUST NOT have an IP address.
Setting ip address to the bridge interface is indeed the most intuitive, but if we do that the vlans cannot communicate with that. And what I am trying to get at is if there is anyway to fix that. Is that a bug? Why is
that setup not feasible. Can't we get the network to be untagged from one port and tagged on the other?
Create a VLAN interface. Create a bridge interface with that VLAN and another physical port as members. Voila, one port tagged, one port untagged.
In the OPNsense context you also need to change two tunables as in the documentation if you use a bridge. If you did not do that, that's a probable reason why your VLANs cannot communicate.
Probably we are miscommunication with each other here. Currently I know that this configuration works:
- eno1.vlan10 -> bridge
- eno2.vlan10 -> bridge
- no router ip, just as a switch (with appropriate firewall setting)
I suspect from your discussion that the following works too, but haven't confirmed:
- eno1.vlan10 -> bridge
- eno2.vlan20 -> bridge
- eno3 (untagged) -> bridge
- no router ip, just as a switch
But the issue that I encounter is in the following setup:
- eno1.vlan10 -> bridge
- eno2.vlan10 -> bridge
- router ip 192.168.1.1/24 on bridge interface
- communication between 192.168.1.2/24 on eno1.vlan10 and 192.168.1.3/24 on eno2.vlan10 works just fine
- communication between 192.168.1.2/24 on eno1.vlan10 and 192.168.1.1/24 on bridge does not work
But you are implying that it should work without a problem, but maybe there are tunables not set. If you are referring to the steps in https://docs.opnsense.org/manual/how-tos/lan_bridge.html#step-six (https://docs.opnsense.org/manual/how-tos/lan_bridge.html#step-six), I have already tried that with no success. My suspicion was that it couldn't communicate because 192.168.1.1/24 was not on the same vlan 10, but you are saying that all traffic coming into eno1.vlan10 gets untagged when it is passed to bridge right? In that case I don't understand why my setup there doesn't work
The layer 2 configuration is correct. Also putting an IP address on the bridge interface is correct. So you assigned the bridge interface to what exactly in Interfaces > Assignments? What are the firewall rules for that interface?
As soon as layer 3 is involved, of course the firewall kicks in.
And yes, you should definitely perform step 6.
To confirm that I am not going crazy, here are screenshots of what you said would be an appropriate configuration and how I can confirm there is no firewall issues.
As you can see there is no connection between the host on the vlan interface and bridge interface static ip. Outside connection to that IP works fine, but from vlan to bridge does not work. vlan to vlan worked last time I checked, but I need to fix my configuration to confirm that again.
Is this virtualized? If yes, I'm out, sorry. Check promiscuous mode and MAC address spoofing on the hypervisor host. If this is not virtualized, what *is* that last screenshot? Never seen anything like this on OPNsense ...
No it is not virtualized. Last screen is a linux terminal of a host on the vlan, not sure what is cryptic about it. It shows I have the correct tag setup and ip within the subnet, but it cannot ping to the gateway. I have more screenshots, but there is a limit of 4 per post, so here are more.
Promiscuous mode was off on all, and I tried turning them all on, but still nothing. I don't know how to do mac spoofing, so any hints on that?
The MAC spoofing and promiscuous mode are only relevant for a hypervisor host.
So the OPNsense is connected via that VLAN interface and a trunk (i.e. tagged) port to a switch and that Linux host is also connected to another trunk (tagged) port on that switch?
You are correct that the frames arrive at the bridge interface untagged when they come in via the VLAN interface. And when they leave via a different VLAN interface they are tagged again.
I'm a bit at the end of suggestions, now, without access to the machine. I'd grab tcpdump and watch what happens on the wire, now.
Could you or any other dev confirm that with the latest opnsense, the vlan is not broken?
Indeed that is the case. Is there another tool for more simple icmp requests?
Two curious things, for some reason I am able to get dhcp address on the machine, and when I do some actions like adding the vlan bridge to an untagged bridged or reverse, then for a brief millisecond the ping goes through in both ways
There were some interfaces with broken VLAN support in FreeBSD. Try ifconfig <phys-interface-with-VLAN-on-top> promisc.
Unfortunately, that did not help. The hardware is an intel Hunsn (can get specifics next week) if that maters in anyway.
Edit: Also to confirm, it is ok that vlan and interface have the same mac addresses right? Running the packet capture, it seems that from the bridge it goes to appropriate vlan interface, but nothing comes back (or go through the other way)
@pmhausen, question, is the fact that I have multiple vlans on other bridges relevant? I.e.:
- bridge1:
- igb0
- igb1
- bridge2:
- igb0.vlan110
- igb1.vlan110
- bridge3:
- igb0.vlan20
- ign1.vlan20
Is it possible that tagged interface is simply going through bridge1 unimpeded, and when I put the ip on the interface it just take the appropriate traffic on that interface, and it just sometimes loops back around from bridge3 to bridge1? I do often see in the firewall log traffic that should be on bridge2 being reported on bridge1.
Are ther appropriate configurations to avoid this? Would using only vlans (i.e. simply disabling bridge1 and related interfaces) fix such an issue?
As soon as the physical interface is a member of some bridge, you cannot have VLANs on that interface. I wrote that early in our conversation. Don't use the untagged interface when you want to use VLANs.
Quote from: Demusman on August 17, 2022, 12:36:17 PM
Don't use the bridge, you'll regret it.
Post pictures of your switch vlan config.
Thanks, I solved it. I had to tag each VLAN to the port that I was using as uplink between the two switches...it seems obvious now but not sure how I missed it earlier :D
Quote from: pmhausen on August 19, 2022, 07:59:11 PM
As soon as the physical interface is a member of some bridge, you cannot have VLANs on that interface. I wrote that early in our conversation. Don't use the untagged interface when you want to use VLANs.
Thank you that indeed solved my problem as well. Hopefully I can still get network boot to work over vlan.
Is it possible to add a check so that when one tries to add a vlan to a bridge where either the untagged interface has an IP or is in a bridge, or the other way around that this warning pops up? This not being to work is not evident, e.g. in OpenWRT you can set up a switch interface with both untagged and tagged interfaces.
That question goes deeper into the code than I can implement. But some more documentation on VLANs and bridging is definitely called for. Glad it's working now.
FreeBSD and OPNsense are simply not a switch - OpenWRT sometimes is depending on the hardware. E.g. FreeBSD does not know port assigned VLANs, actually it doesn't know VLANs at all, only tagged imterfaces. So one has to resort to getting creative with the bridge interface. It's like Cisco router vs. Cisco switch.
General recommendation: use a fast enough trunk port, better yet LACP, to connect to a switch, do everything else on the switch. So called router-on-a-stick architecture.
Hey All,
Thanks for the good explanation Patrick.
Using this post+other material, I used a bridged setup to expose one of my VMs to the internet.
I wrote a blog post detailing my setup and I hope it helps other people =)
OPNsense - WAN Bridging for Public IP Addressing
https://blog.infoitech.co.uk/opnsense-wan-bridging-public-address/ (https://blog.infoitech.co.uk/opnsense-wan-bridging-public-address/)
Kind regards.
@markh0ppus Not sure why you would need a bridge for that. All you want to do is assign a public IPv4 address to a host in the OPNsense LAN, correct?
You can do that by e. g. enabling dynamic gateway policy on the LAN interface and creating a static route for the desired public IPv4 address (/32), pointing to the dynamic LAN "gateway" (just the interface in fact). This also enables ARP for this IP address, despite it not being in the actual LAN subnet.
On the host itself, configure the interface with the public IPv4 address (/32). The gateway is the OPNsense LAN address.
You might also have to add a proxy ARP alias for said IPv4 address to the OPNsense WAN, depending on how your ISP routes the /29.
Cheers
Maurice