Looking for some guidance on a new network setup

[eight-molecules]

hi all, i'm looking for some help in how i can set up a new network. i'm happy to take links and guides, but i've gotten to the halfway point of some connectivity and all i've got is that it's not exactly what i'm looking for and i need a little help with my configs. this is what i'd like to do:

2 machines running 2 OPNSense VMs in HA, with OPNSense taking direct control of a 4 port NIC. both hosts have WAN on port 1 and a LAG connection on 2-4 into a GS728TP. the GS728TP is hosting these two OPNSense VM host machines through their native ethernet port, along with 5 (and growing) machines for storage and compute. all these boxes are running Ubuntu Server and serving VMs with libvirt/Cockpit.

i also have a private network that can exist inside the OPNSense hardware network, but i'd prefer it didn't. this is my google mesh network, and it should provide a passthrough secondary WAN connection to one of the hosts. i need to be able to securely access the WebGUI from this network so i can manage it. i would also like to be able to address VMs inside the cluster from a domain that i manage using something like ddclient and AWS to manage its connectivity.

one limitation is my main access to setup OPNSense is from the private network serving a DHCP address to the WAN side connection of the one OPNSense VM. for security i want all my traffic directly to the hardware to be buffered through the OPNSense VMs. these are all pretty new things for me, i've been doing SDN container stuff for a while and everything got more simple and yet more complicated since the last time i helped string together a hardware network. i drew a diagram, i hope this makes sense, and i hope someone can help me get squared away.

[Strator]

How do you manage the VM host? How do you manage the switch? You must have some management VLAN for it, right? I would manage the OPNsense VMs on the same VLAN instead of trying to get into them from their WAN side.

[eight-molecules]

WAN side security: Google Workspace SSO on ingress to my network. if i have auth i have access to the various IPs. this is long term, short term is turning the power off between runs.

VMs: Cockpit and KVM/QEMU (on top of libvirt), currently looking at additional tools. i want to manage the VM host by addressing a subdomain resolved by my OPNSense host (e.g. host-0.veryscary.link, host-1.veryscary.link). to ensure some semblance of vendor segregation i want my hosts to be limited to having access over their onboard ethernet if the 350-T4s are up and being managed by OPNSense. if not i will carry a keyboard and monitor over and diagnose, but the goal is to minimize this requirement (for remote deployments). Hosts are also on a management VLAN, and VMs on the hosts are in their own management VLAN. VLANs everywhere.

Switch: Netgear managed switch on a VLAN behind the main box separated from the hosts.

I'd like to get in from the WAN side because I have access from the WAN side. i can already start the process by plugging my private network directly into my host's WAN port and address it by hostname to configure it, or open a graphical session over the network using Cockpit. my ideal state is to have all HTTP access go out through the aforementioned VM VLANs, through OPNSense, and out the other side. securing remote access is a whole other thing i need to reach and for now i'm pulling the cluster offline at the end of each day. i know a lot of what i need to do long term to bring the cluster to a secure state and i'd like to have a process that allows for remote configuration and maintenance with any host count.

[Strator]

When you go VM and VLAN, there is a psychical layer and logical layer. The way I approach it, I design my network in the logical layer first and than I try to implement it in the physical layer. The physical layer usually have some limitations that force me to make adjustments to the logical layer.

If I wanted to get any sound advice about my network setup, I would present both views of my network, and include the capabilities of the involved devices.

[eight-molecules]

resolved my own issue by breaking my process down further. wan only enables the web gui on wan, a neat trick since you don't actually need multiple cables/nics for wan and lan if you try hard enough.

have access, am unblocked