1
24.1 Production Series / VLAN routing improvement
« on: May 26, 2024, 02:45:37 pm »
I have some overkill of home network.
Base is a UDM SE, 24Port L3 switch and a L2 10GbE aggregation switch. Connected to it is an AMD (Ryzen 4650 Pro, dual Intel 10GbE NIC, 82599ES) and a xcp-ng hosting MinisForum MS-01 (12900H with 32GB RAM and dual SFP+ Intel x710); both system are connected via DAC to the aggregation switch. That switch then via DAC to the UDM SE.
On the xcp-ng host I have a virtual OPNsense 24.1.7-4 connected to one SFP+ port and three Debian 12 VM for testing in two different VLAN. Those VLAN are also known to the OPNsense on the host. No WAN.
The firewall rule is a floating one. IPv4 for LAN, OPT1 and OPT2 interface; allow all traffic in any direction (to start with)
Goal is to have a quick storage box connected to various systems (e.g. the xcp-ng host, and later Mac Studio).
The first pic shows the speed of a Debian VM on the xcp-ng to the physical box via iperf3. Nice speed of 9.25Gbits/sec. Low number of retries. No complains and quite close to the thoretical 10Gbits/sec. Understand there is overhead involved.
On the second pic I have three measurements.
1) within the same VLAN/subnet on two Debian VM. 25.9 Gbits/sec and huge retry count (~26000).
2) between two Debian in different VLAN/subnets on the same physical host with 5.19 Gbits/sec and also big retry count via a virtual OPNsense VM (~8500)
3) from a Debian VM in different VLAN/subnet to physical unraid box (via UDM SE); routing is with 3.99 Gbits/sec below expectation. Retry 42
Question 1) Is the high retry in iperf3 for case 1 and 2 above something to worry ? It's a bit academic as pure software but on the other side: still no idea where those retry numbers come from. Any chance I can find that in OPNsense logs ?
Question 2) What would be a better network setup to achieve a high speed storage network. The current OPNsense as VM is a trial to see if I can configure it somehow (and seems working basically). Might buy a second MS-01 as dedicated physical router since the UDM SE is on its limit.
Base is a UDM SE, 24Port L3 switch and a L2 10GbE aggregation switch. Connected to it is an AMD (Ryzen 4650 Pro, dual Intel 10GbE NIC, 82599ES) and a xcp-ng hosting MinisForum MS-01 (12900H with 32GB RAM and dual SFP+ Intel x710); both system are connected via DAC to the aggregation switch. That switch then via DAC to the UDM SE.
On the xcp-ng host I have a virtual OPNsense 24.1.7-4 connected to one SFP+ port and three Debian 12 VM for testing in two different VLAN. Those VLAN are also known to the OPNsense on the host. No WAN.
The firewall rule is a floating one. IPv4 for LAN, OPT1 and OPT2 interface; allow all traffic in any direction (to start with)
Goal is to have a quick storage box connected to various systems (e.g. the xcp-ng host, and later Mac Studio).
The first pic shows the speed of a Debian VM on the xcp-ng to the physical box via iperf3. Nice speed of 9.25Gbits/sec. Low number of retries. No complains and quite close to the thoretical 10Gbits/sec. Understand there is overhead involved.
On the second pic I have three measurements.
1) within the same VLAN/subnet on two Debian VM. 25.9 Gbits/sec and huge retry count (~26000).
2) between two Debian in different VLAN/subnets on the same physical host with 5.19 Gbits/sec and also big retry count via a virtual OPNsense VM (~8500)
3) from a Debian VM in different VLAN/subnet to physical unraid box (via UDM SE); routing is with 3.99 Gbits/sec below expectation. Retry 42
Question 1) Is the high retry in iperf3 for case 1 and 2 above something to worry ? It's a bit academic as pure software but on the other side: still no idea where those retry numbers come from. Any chance I can find that in OPNsense logs ?
Question 2) What would be a better network setup to achieve a high speed storage network. The current OPNsense as VM is a trial to see if I can configure it somehow (and seems working basically). Might buy a second MS-01 as dedicated physical router since the UDM SE is on its limit.