Networking Question - Isolating VM's on network

[talespin]

Hi Everyone,

My Setup:
   Internet > ISP Router(into OPNSense WAN Port) > OPN Sense Router(one WAN port and 4 Lan port(LAN bridged)) > All devices(Server, AP and other devices on network into those 4 LAN Ports)

What I have :
single home server Host Debian KVM, it hosts multiple VM's. There will be client machines that will be connecting to these VM's over LAN. Two of VM's are gaming VM.
Currently OPNSense is assigning IP to VM's.

Objective:
   1. I want to be able to isolate individual VM's on network so they don't see other VM's on LAN or communicate with host on network.
      Example
      VM1 should not see/access VM2,3,4...
      VM2 should be able to see VM4 only.

   2. I don't want to use virtual switch on host for performance reason. I'll be using network card with SRIOV functionality assigning virtual functions to each VM or maybe passing physical port to each VM.


1. I don't know much about networking and need guidance on how to achieve objective. What I have learned so far is that I need to use subnet/VLAN to be able to isolate individusl VM's. I could create 4 VLAN on each of the 4 ports with different subnet but then how do I connect VM's to these separate VLAN's/subnet.

2. What is the difference between these from performance perspective?
Software bridge on host with single physical port.
vs
Passing through physical ports to each VM, using a network card with multiple ports supporting IOMMU.
vs
Assigning virtual functions to VM's SRIOV.

Thank you

[meyergru]

I can only tell you how I am doing that kind of thing with PVE (which also uses KVM, so this should be similar enough):

I have one bridge vmbr0 connected to a physical ethernet adapter that connects to the "real" network and it also has the IP of the PVE host.
A second bridge interface vmbr1, is VLAN-aware and is purely virtual, in that it has no physical NICs attached to it at all.
For each VM, I create a separate VLAN X with a an associated subnet, say 10.0.X.0/24. The VM X itself has 10.0.X.2/24 as its IP address and 10.0.x.1 as gateway and DNS server.

I use one OpnSense VM that has its virtio "WAN" attached to vmbr0 and thus gets an IP on my LAN. It also has a virtio "VLANBRIDGE" interface, off of which I define "VLANX" interfaces accordingly with VLAN X and IP address 10.0.x.1/24.

Each of the VLANXs is allowed to access the internet, but not RFC1918, so it cannot access either LAN devices or other VMs.

On the other hand, I can create rules that allow access from the "WAN" (aka LAN) to each VM (or the whole 10.0.0.0/16 range).

You can use such a setup in a datacenter or at home, like described here: https://forum.opnsense.org/index.php?topic=44159.0

[talespin]

I can only tell you how I am doing that kind of thing with PVE (which also uses KVM, so this should be similar enough):

I have one bridge vmbr0 connected to a physical ethernet adapter that connects to the "real" network and it also has the IP of the PVE host.
A second bridge interface vmbr1, is VLAN-aware and is purely virtual, in that it has no physical NICs attached to it at all.
For each VM, I create a separate VLAN X with a an associated subnet, say 10.0.X.0/24. The VM X itself has 10.0.X.2/24 as its IP address and 10.0.x.1 as gateway and DNS server.

I use one OpnSense VM that has its virtio "WAN" attached to vmbr0 and thus gets an IP on my LAN. It also has a virtio "VLANBRIDGE" interface, off of which I define "VLANX" interfaces accordingly with VLAN X and IP address 10.0.x.1/24.

Each of the VLANXs is allowed to access the internet, but not RFC1918, so it cannot access either LAN devices or other VMs.

On the other hand, I can create rules that allow access from the "WAN" (aka LAN) to each VM (or the whole 10.0.0.0/16 range).

You can use such a setup in a datacenter or at home, like described here: https://forum.opnsense.org/index.php?topic=44159.0

Thank you for sharing your approach.

From what I understand, you are using virtual switch (software) and subnet. I want to explore hardware option before using virtual switch to reduce latency for gaming VM's.

[meyergru]

You are assuming that "hardware" must be faster than "software". That is not generally true.

If OPNsense itself runs as a VM on that host, the traffic has to pass through a VM anyway, so adding physical NICs and an external switch only adds another network path.

Only if OPNsense is a separate physical firewall does the traffic need to leave the host in the first place.

In either case, I would start with virtio + a Linux bridge and only change that if measurements show an actual problem.

[talespin]

Hi,

You are assuming that "hardware" must be faster than "software". That is not generally true.

If OPNsense itself runs as a VM on that host, the traffic has to pass through a VM anyway, so adding physical NICs and an external switch only adds another network path.

You are correct. If traffic is within server, virtual switch would be faster vs any hardware based approach for VM to VM traffic.

In my case client machines will be connecting to gaming VM over LAN, so traffic has to leave VM Host server.

Sorry I should have mentioned.
OPNSense router is a separate system(M720q) with a 4 port NIC.

In either case, I would start with virtio + a Linux bridge and only change that if measurements show an actual problem.

I'll definitely try this, right now I am gathering information and exploring my options so that I can compare what works best.

Thank you