Guys,
After 4 weeks of wrestling with several tutorials I gave up. I need some help. I have a barebone machine with 4 core, 8GB memory, 64GB SSD and 4 NIC's. On this I installed proxmox. In proxmox I have a bridge on ensf0 and created a linux bridge on ensf1-3, not "VLAN aware". This should work as a common switch.
Next step I installed OPNsense and created a LAN and WAN on the bridges. I added a DHCP server on the LAN and created some rules in the firewall to play with.
This works fine. All firewall rules are obeyed and the switch based on the 3 NIC's works like a charm.
Next step I created two VLAN's (33 and 44) on the same bridge as the LAN, both having their own DCHP. I included in both vlans the rule that they can access everything in their own vlan and only 33 is also allowed to reach ip-addresses in 44. Seemed a reasonable usecase to me.
Third step is the addition of a netgear managed switch which has port 5 connected to ensf2. The switch is configured as:
1 2 3 4 5
vlan 1 U
vlan 33 U U T
vlan 44 U U T
And the PVID
1 vlan 33
2 vlan 33
3 vlan 44
4 vlan 44
5 vlan 1
I connect my laptop to port 1 on this switch. The DHCP is not found, no IP address is assigned. If I assign a fixed IP in the range of VLAN 33 I still cannot access anything. I looks like the VLAN's are not on the ensf1-3 switch in proxmox.
So flipping arround VLAN-aware, disabling the LAN DHCP, trying different modes of the netgear switch, reinstalling opnsense for the 5th time, trying to create the bridge in OPNsense, ....
The reason why I want the switch in proxmox is that 3 cables are running from my entry point in the house to 3 different rooms. I want to have the vlans in all rooms. I attached the physical network.
Who can help me out getting this operational
Is there a reason you're using Proxmox instead of just running OPNSense bare metal? Your diagram shows managed switches, so you can let them break out the VLANs to separate ports.
Got it fully operational.
What I did was create a linux bridge in ProxMox as vmbr1 where all NIC's except nr 1 where added, just look for the tutorial on ProxMox and Linux bridge. This will respond as a "kinda L3 switch" since the cables in my house are connecting to a managed L2 switch.
Next installed OPNsense on a VM and added the vmbr1 as my LAN NIC. My network was responding on all switches.
In OPNsense I build my 4 basic VLANs and had to add 3 mandatory VLANs from my ISP to have all connections operational. On the OPNsense firewall I arranged the separation of the VLANs, so they can't interact without help of additional layer of functionality. And I blocked the https access to the firewall from all VLAN's except 1 (that's my administrator vlan). I created a VM on my laptop with a VPN to that latter VLAN and can now access all thru a secured box without needing to expose my administator VLAN on a port of a switch. And with a trunc a connected the set of vlans to a NUC that runs a hypervisor. Here I add dockers to any of the VLAN's.
Have only one thing to investigate: can I somehow add the baremetal machine hosting proxmox and OPNsense to my administrator network. My guts say no, you can't. But let's explore.
I'm happy.