Sanity check on linux bridge setup for upstream inter-VLAN routing

[OPNenthu]

I migrated my desktop PC to Linux and also migrated some VirtualBox VMs there.  My goal is to expose two VLANs from OPNsense to the Linux host, on two separate linux bridges, so that the host and guest VMs can attach to either VLAN as needed and also maintain traffic isolation between them (no host routing).  I want OPNsense to handle the inter-VLAN routing and enforce its policies.

The host has a 2.5GbE Intel i226-v NIC that is connected via a trunk switch port configured as native=30(CLEAR) and tagged=20(VPN).  The host is to use 'br0' which carries the untagged traffic.  Guest VMs can attach to either 'br0' (for clear internet) or 'br20' (for VPN gateway).  OPNsense policies allow clients on VLAN 20 to reach local services on VLAN 30.

After some experimentation and failures, the best working setup I came up with is this:

Code Select Expand

$ ip a
...
3: enp10s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 24:xx:xx:xx:xx:cd brd ff:ff:ff:ff:ff:ff
4: enp10s0.20@enp10s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br20 state UP group default qlen 1000
    link/ether 24:xx:xx:xx:xx:cd brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 24:xx:xx:xx:xx:cd brd ff:ff:ff:ff:ff:ff
    inet 172.21.30.100/24 brd 172.21.30.255 scope global dynamic noprefixroute br0
       valid_lft 86118sec preferred_lft 86118sec
    inet6 2601:xx:xxxx:6db3:e7f7:39a6:1d2d:bed4/64 scope global temporary dynamic
       valid_lft 86371sec preferred_lft 85760sec
    inet6 2601:xx:xxxx:6db3:xxxx:xxxx:xxxx:9dca/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86371sec preferred_lft 86371sec
    inet6 fe80::xxxx:xxxx:xxxx:fb89/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
6: br20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a2:xx:xx:xx:xx:5a brd ff:ff:ff:ff:ff:ff
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:xx:xx:xx:xx:76 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

(virbr0 is created by VirtualBox for its NAT networking- I don't manage it.)

Using NetworkManager / nmcli, I created br0 which has the NIC (enp10s0) as a slave port.  br0 also has IP addresses for the host itself to access VLAN 30.

I then created a VLAN sub-interface (enp10s0.20) to handle tagging on VLAN 20 and made this a slave port on br20.  I left br20 unconfigured because the host doesn't use it and any guest VMs attached to it can configure themselves with DHCP / SLAAC.  This bridge should hopefully make tagging transparent to the VMs and they can just pass untagged frames internally.

I also disabled IP forwarding globally via sysctl config:

Code Select Expand

$ cat /etc/sysctl.d/999-disable-ip-forwarding.conf
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0

... and confirmed that no host route exists for VLAN 20:

Code Select Expand

$ ip r
default via 172.21.30.1 dev br0 proto dhcp src 172.21.30.100 metric 425
172.21.30.0/24 dev br0 proto kernel scope link src 172.21.30.100 metric 425
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

$ ip -6 r
2601:xx:xxxx:6db3::/64 dev br0 proto ra metric 425 pref medium
fe80::/64 dev br0 proto kernel metric 1024 pref medium
default via fe80::xxxx:xxxx:xxxx:39a0 dev br0 proto ra metric 425 pref medium

So far so good and everything "works" as expected.  I have a guest VM in VirtualBox that is acting as a NAS on VLAN 30 and another guest VM that is acting as an SMB client on VLAN 20.  The client's internet is going through the VPN gateway and online speedtest results look great- full speed achieved with an 'A' score on the bufferbloat Waveform test.  From OPNsense logs I can see the inter-VLAN routing is taking place when I transfer a file from NAS->client:

You cannot view this attachment.

I observe a couple issues, however.

The first is not serious and I can live with it.  It's that the host bridge br0 takes some time after system boot to get its IP address.  When I was using the physical interface directly, DHCP would already be done by the time the desktop booted up.  With the bridge it takes an additional half a minute after logging on to get the IPs configured.  I expect SLAAC to have some delay because of RA intervals, but DHCP delay seems odd.

The second issue is that I am seeing high retransmit counts and small TCP congestion windows in iperf3 between the two VMs.  They are sharing a physical link up to the switch, but it should be full-duplex.  This is the iperf3 result from client to server VM:

Code Select Expand

$ iperf3 -c 172.21.30.108
Connecting to host 172.21.30.108, port 5201
[  5] local 172.21.20.130 port 40986 connected to 172.21.30.108 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   199 MBytes  1.67 Gbits/sec  436    277 KBytes       
[  5]   1.00-2.00   sec   211 MBytes  1.77 Gbits/sec   74    339 KBytes       
[  5]   2.00-3.00   sec   252 MBytes  2.12 Gbits/sec  174    349 KBytes       
[  5]   3.00-4.00   sec   236 MBytes  1.98 Gbits/sec  116    419 KBytes       
[  5]   4.00-5.00   sec   218 MBytes  1.82 Gbits/sec  131    290 KBytes       
[  5]   5.00-6.00   sec   206 MBytes  1.73 Gbits/sec   56    363 KBytes       
[  5]   6.00-7.00   sec   230 MBytes  1.93 Gbits/sec  161    356 KBytes       
[  5]   7.00-8.00   sec   199 MBytes  1.67 Gbits/sec   70    370 KBytes       
[  5]   8.00-9.00   sec   199 MBytes  1.67 Gbits/sec   51    358 KBytes       
[  5]   9.00-10.00  sec   188 MBytes  1.57 Gbits/sec   99    338 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.09 GBytes  1.79 Gbits/sec  1368             sender
[  5]   0.00-10.00  sec  2.08 GBytes  1.79 Gbits/sec                  receiver

It's a similar story in the opposite direction.

I can accept this for my needs, but I am curious what's causing it and if I misconfigured something.  I suspect fragmentation, maybe due to the VLAN tag overhead (?) but I'm not sure how to confirm.  All interfaces are using 1500 MTU as confirmed in linux.

My second question is regarding the architecture itself: is there anything that I overlooked which might come back to bite me?  Did I open myself to VPN leaks from the br20 clients?

TIA!