[SOLVED] OPNsense on Proxmox with linux bridge switch - VLANs not working

[talldragon]

Guys,
After 4 weeks of wrestling with several tutorials I gave up. I need some help. I have a barebone machine with 4 core, 8GB memory, 64GB SSD and 4 NIC's. On this I installed proxmox. In proxmox I have a bridge on ensf0 and created a linux bridge on ensf1-3, not "VLAN aware". This should work as a common switch.
Next step I installed OPNsense and created a LAN and WAN on the bridges. I added a DHCP server on the LAN and created some rules in the firewall to play with.
This works fine. All firewall rules are obeyed and the switch based on the 3 NIC's works like a charm.

Next step I created two VLAN's (33 and 44) on the same bridge as the LAN, both having their own DCHP. I included in both vlans the rule that they can access everything in their own vlan and only 33 is also allowed to reach ip-addresses in 44. Seemed a reasonable usecase to me.

Third step is the addition of a netgear managed switch which has port 5 connected to ensf2. The switch is configured as:
              1     2     3      4     5
vlan 1                                 U
vlan 33    U     U                  T
vlan 44                 U     U     T
And the PVID
1   vlan 33
2   vlan 33
3   vlan 44
4   vlan 44
5   vlan 1

I connect my laptop to port 1 on this switch. The DHCP is not found, no IP address is assigned. If I assign a fixed IP in the range of VLAN 33 I still cannot access anything. I looks like the VLAN's are not on the ensf1-3 switch in proxmox.

So flipping arround VLAN-aware, disabling the LAN DHCP, trying different modes of the netgear switch, reinstalling opnsense for the 5th time, trying to create the bridge in OPNsense, ....

The reason why I want the switch in proxmox is that 3 cables are running from my entry point in the house to 3 different rooms. I want to have the vlans in all rooms. I attached the physical network.


Who can help me out getting this operational

[CJ]

Is there a reason you're using Proxmox instead of just running OPNSense bare metal?  Your diagram shows managed switches, so you can let them break out the VLANs to separate ports.

[talldragon]

Got it fully operational.
What I did was create a linux bridge in ProxMox as vmbr1 where all NIC's except nr 1 where added, just look for the tutorial on ProxMox and Linux bridge. This will respond as a "kinda L3 switch" since the cables in my house are connecting to a managed L2 switch.
Next installed OPNsense on a VM and added the vmbr1 as my LAN NIC. My network was responding on all switches.
In OPNsense I build my 4 basic VLANs and had to add 3 mandatory VLANs from my ISP to have all connections operational. On the OPNsense firewall I arranged the separation of the VLANs, so they can't interact without help of additional layer of functionality. And I blocked the https access to the firewall from all VLAN's except 1 (that's my administrator vlan). I created a VM on my laptop with a VPN to that latter VLAN and can now access all thru a secured box without needing to expose my administator VLAN on a port of a switch. And with a trunc a connected the set of vlans to a NUC that runs a hypervisor. Here I add dockers to any of the VLAN's.
Have only one thing to investigate: can I somehow add the baremetal machine hosting proxmox and OPNsense to my administrator network. My guts say no, you can't. But let's explore.

I'm happy.