I migrated my desktop PC to Linux and also migrated some VirtualBox VMs there. My goal is to expose two VLANs from OPNsense to the Linux host, on two separate linux bridges, so that the host and guest VMs can attach to either VLAN as needed and also maintain traffic isolation between them (no host routing). I want OPNsense to handle the inter-VLAN routing and enforce its policies.
The host has a 2.5GbE Intel i226-v NIC that is connected via a trunk switch port configured as native=30(CLEAR) and tagged=20(VPN). The host is to use 'br0' which carries the untagged traffic. Guest VMs can attach to either 'br0' (for clear internet) or 'br20' (for VPN gateway). OPNsense policies allow clients on VLAN 20 to reach local services on VLAN 30.
After some experimentation and failures, the best working setup I came up with is this:
$ ip a
...
3: enp10s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
link/ether 24:xx:xx:xx:xx:cd brd ff:ff:ff:ff:ff:ff
4: enp10s0.20@enp10s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br20 state UP group default qlen 1000
link/ether 24:xx:xx:xx:xx:cd brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 24:xx:xx:xx:xx:cd brd ff:ff:ff:ff:ff:ff
inet 172.21.30.100/24 brd 172.21.30.255 scope global dynamic noprefixroute br0
valid_lft 86118sec preferred_lft 86118sec
inet6 2601:xx:xxxx:6db3:e7f7:39a6:1d2d:bed4/64 scope global temporary dynamic
valid_lft 86371sec preferred_lft 85760sec
inet6 2601:xx:xxxx:6db3:xxxx:xxxx:xxxx:9dca/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 86371sec preferred_lft 86371sec
inet6 fe80::xxxx:xxxx:xxxx:fb89/64 scope link noprefixroute
valid_lft forever preferred_lft forever
6: br20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether a2:xx:xx:xx:xx:5a brd ff:ff:ff:ff:ff:ff
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:xx:xx:xx:xx:76 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
(virbr0 is created by VirtualBox for its NAT networking- I don't manage it.)
Using NetworkManager / nmcli, I created br0 which has the NIC (enp10s0) as a slave port. br0 also has IP addresses for the host itself to access VLAN 30.
I then created a VLAN sub-interface (enp10s0.20) to handle tagging on VLAN 20 and made this a slave port on br20. I left br20 unconfigured because the host doesn't use it and any guest VMs attached to it can configure themselves with DHCP / SLAAC. This bridge should hopefully make tagging transparent to the VMs and they can just pass untagged frames internally.
I also disabled IP forwarding globally via sysctl config:
$ cat /etc/sysctl.d/999-disable-ip-forwarding.conf
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0
... and confirmed that no host route exists for VLAN 20:
$ ip r
default via 172.21.30.1 dev br0 proto dhcp src 172.21.30.100 metric 425
172.21.30.0/24 dev br0 proto kernel scope link src 172.21.30.100 metric 425
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
$ ip -6 r
2601:xx:xxxx:6db3::/64 dev br0 proto ra metric 425 pref medium
fe80::/64 dev br0 proto kernel metric 1024 pref medium
default via fe80::xxxx:xxxx:xxxx:39a0 dev br0 proto ra metric 425 pref medium
So far so good and everything "works" as expected. I have a guest VM in VirtualBox that is acting as a NAS on VLAN 30 and another guest VM that is acting as an SMB client on VLAN 20. The client's internet is going through the VPN gateway and online speedtest results look great- full speed achieved with an 'A' score on the bufferbloat Waveform test. From OPNsense logs I can see the inter-VLAN routing is taking place when I transfer a file from NAS->client:
inter-vlan-ok.png
I observe a couple issues, however.
The first is not serious and I can live with it. It's that the host bridge br0 takes some time after system boot to get its IP address. When I was using the physical interface directly, DHCP would already be done by the time the desktop booted up. With the bridge it takes an additional half a minute after logging on to get the IPs configured. I expect SLAAC to have some delay because of RA intervals, but DHCP delay seems odd.
The second issue is that I am seeing high retransmit counts and small TCP congestion windows in iperf3 between the two VMs. They are sharing a physical link up to the switch, but it should be full-duplex. This is the iperf3 result from client to server VM:
$ iperf3 -c 172.21.30.108
Connecting to host 172.21.30.108, port 5201
[ 5] local 172.21.20.130 port 40986 connected to 172.21.30.108 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 199 MBytes 1.67 Gbits/sec 436 277 KBytes
[ 5] 1.00-2.00 sec 211 MBytes 1.77 Gbits/sec 74 339 KBytes
[ 5] 2.00-3.00 sec 252 MBytes 2.12 Gbits/sec 174 349 KBytes
[ 5] 3.00-4.00 sec 236 MBytes 1.98 Gbits/sec 116 419 KBytes
[ 5] 4.00-5.00 sec 218 MBytes 1.82 Gbits/sec 131 290 KBytes
[ 5] 5.00-6.00 sec 206 MBytes 1.73 Gbits/sec 56 363 KBytes
[ 5] 6.00-7.00 sec 230 MBytes 1.93 Gbits/sec 161 356 KBytes
[ 5] 7.00-8.00 sec 199 MBytes 1.67 Gbits/sec 70 370 KBytes
[ 5] 8.00-9.00 sec 199 MBytes 1.67 Gbits/sec 51 358 KBytes
[ 5] 9.00-10.00 sec 188 MBytes 1.57 Gbits/sec 99 338 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.09 GBytes 1.79 Gbits/sec 1368 sender
[ 5] 0.00-10.00 sec 2.08 GBytes 1.79 Gbits/sec receiver
It's a similar story in the opposite direction.
I can accept this for my needs, but I am curious what's causing it and if I misconfigured something. I suspect fragmentation, maybe due to the VLAN tag overhead (?) but I'm not sure how to confirm. All interfaces are using 1500 MTU as confirmed in linux.
My second question is regarding the architecture itself: is there anything that I overlooked which might come back to bite me? Did I open myself to VPN leaks from the br20 clients?
TIA!