VLAN routing improvement

[ChristianVirtual]

I have some overkill of home network.

Base is a UDM SE, 24Port L3 switch and a L2 10GbE aggregation switch. Connected to it is an AMD (Ryzen 4650 Pro, dual Intel 10GbE NIC, 82599ES) and a xcp-ng hosting MinisForum MS-01 (12900H with 32GB RAM and dual SFP+ Intel x710); both system are connected via DAC to the aggregation switch. That switch then via DAC to the UDM SE.

On the xcp-ng host I have a virtual OPNsense 24.1.7-4 connected to one SFP+ port and three Debian 12 VM for testing in two different VLAN. Those VLAN are also known to the OPNsense on the host. No WAN.

The firewall rule is a floating one. IPv4 for LAN, OPT1 and OPT2 interface; allow all traffic in any direction (to start with)

Goal is to have a quick storage box connected to various systems (e.g. the xcp-ng host, and later Mac Studio).

The first pic shows the speed of a Debian VM on the xcp-ng to the physical box via iperf3. Nice speed of 9.25Gbits/sec. Low number of retries. No complains and quite close to the thoretical 10Gbits/sec. Understand there is overhead involved.

On the second pic I have three measurements.
1) within the same VLAN/subnet on two Debian VM. 25.9 Gbits/sec and huge retry count (~26000).
2) between two Debian in different VLAN/subnets on the same physical host with 5.19 Gbits/sec and also big retry count via a virtual OPNsense VM (~8500)
3) from a Debian VM in different VLAN/subnet to physical unraid box (via UDM SE); routing is with 3.99 Gbits/sec below expectation. Retry 42


Question 1) Is the high retry in iperf3 for case 1 and 2 above something to worry ? It's a bit academic as pure software but on the other side: still no idea where those retry numbers come from. Any chance I can find that in OPNsense logs ?

Question 2) What would be a better network setup to achieve a high speed storage network. The current OPNsense as VM is a trial to see if I can configure it somehow (and seems working basically). Might buy a second MS-01 as dedicated physical router since the UDM SE is on its limit.

[ChristianVirtual]

No one has an idea ?

Is OPNsense a good router for 5Gbits/s ~ 10Gbits/s or should I better look for other tools like fd.io / tnsr ?

[cookiemonster]

cases like these I see them better on a schematic so I might be getting it wrong. High retries normally relate to physical rather than config but drivers can play a part. You also have virtual nic or passthrough in play (Honestly this is an assumption, I have basic understanding only of xcp-ng. I normally use esxi or proxmox).