DEC 4040 / iperf3 10G single threat question

[numiralofe]

hi,

I have a DEC 4040 and recently we have upgraded our internet connection to 10G but i am having some issues understanding the results that i am getting, and would like to ask some help understanding them.

Setup is as follows:

Internet Router ( 10G ) --> OpnSense DEC 4040 ( ax1 port / SFP+ module ) --> Mikrotik Switch ( ax0 port ) --> Internal LAN ( there are no vlans )


ax0 and ax1 ports are both detected as 10Gbase

Code Select Expand


ax0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN (lan)
options=4e0032b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether f4:90:ea:00:bb:2a
inet 192.168.12.1 netmask 0xffffff00 broadcast 192.168.12.255
inet 192.168.12.254 netmask 0xffffff00 broadcast 192.168.12.255
inet6 fe80::f690:eaff:fe00:bb2a%ax0 prefixlen 64 scopeid 0x7
media: Ethernet autoselect (10GBase-SFI <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


Code Select Expand


ax1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (opt2)
options=4e0032b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether f4:90:ea:00:bb:2b
inet 192.168.2.206 netmask 0xffffff00 broadcast 192.168.2.255
inet 192.168.2.200 netmask 0xffffff00 broadcast 192.168.2.255
inet6 fe80::f690:eaff:fe00:bb2b%ax1 prefixlen 64 scopeid 0x8
media: Ethernet autoselect (10GBase-SFI <full-duplex,rxpause,txpause>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>




my question is:

if i run iperf server on a local network vm and a single iperf3 threat from opnsense into the vm I can get almost 10G

opnsense --> local network vm

Code Select Expand


root@opnsense-hw:~ # iperf3 -c 192.168.12.122 -p 5001
Connecting to host 192.168.12.122, port 5001
[  5] local 192.168.12.1 port 10549 connected to 192.168.12.122 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.04   sec  1.02 GBytes  8.41 Gbits/sec    0   3.00 MBytes       
[  5]   1.04-2.00   sec  1.06 GBytes  9.41 Gbits/sec    0   3.00 MBytes       
[  5]   2.00-3.04   sec  1.14 GBytes  9.41 Gbits/sec    0   3.00 MBytes       
[  5]   3.04-4.00   sec  1.05 GBytes  9.41 Gbits/sec    0   3.00 MBytes       
[  5]   4.00-5.00   sec  1.09 GBytes  9.41 Gbits/sec    0   3.00 MBytes


nevertheless the other way around ( from vm into iperf server running on opnsense ) i won't get the same speed...

local network vm --> opnsense

Code Select Expand


root@debian-12:~$ iperf3 -c 192.168.2.1 -p 5001
Connecting to host 192.168.2.1, port 5001
[  5] local 192.168.12.122 port 60912 connected to 192.168.2.1 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   475 MBytes  3.98 Gbits/sec   32   2.37 MBytes       
[  5]   1.00-2.00   sec   510 MBytes  4.28 Gbits/sec    0   2.56 MBytes       
[  5]   2.00-3.00   sec   511 MBytes  4.29 Gbits/sec    0   2.71 MBytes       
[  5]   3.00-4.00   sec   512 MBytes  4.30 Gbits/sec    0   2.84 MBytes       
[  5]   4.00-5.00   sec   515 MBytes  4.32 Gbits/sec    0   2.93 MBytes       
[  5]   5.00-6.00   sec   511 MBytes  4.29 Gbits/sec    1   2.22 MBytes 


again from opnsense into the internet router i can also run a single iperf3 thread at 10G

opnsense --> internet fiber router

Code Select Expand


root@opnsense-hw:~ # iperf3 -c 192.168.2.1 -p 5001
Connecting to host 192.168.2.1, port 5001
[  5] local 192.168.2.206 port 45143 connected to 192.168.2.1 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.06   sec  1.16 GBytes  9.40 Gbits/sec    0   3.00 MBytes       
[  5]   1.06-2.06   sec  1.09 GBytes  9.41 Gbits/sec    0   3.00 MBytes       
[  5]   2.06-3.06   sec  1.10 GBytes  9.41 Gbits/sec    0   3.00 MBytes 


but from any vm behind opnsense any single iperf3 thread to the internet router gets capped at 4G...

local network vm --> opnsense --> fiber router

Code Select Expand


root@debian-12:~$ iperf3 -c 192.168.2.1 -p 5001
Connecting to host 192.168.2.1, port 5001
[  5] local 192.168.12.122 port 34198 connected to 192.168.2.1 port 5001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   421 MBytes  3.53 Gbits/sec   60   2.32 MBytes       
[  5]   1.00-2.00   sec   514 MBytes  4.31 Gbits/sec    0   2.51 MBytes       
[  5]   2.00-3.00   sec   512 MBytes  4.30 Gbits/sec    0   2.67 MBytes       
[  5]   3.00-4.00   sec   515 MBytes  4.32 Gbits/sec    0   2.79 MBytes       
[  5]   4.00-5.00   sec   516 MBytes  4.33 Gbits/sec    0   2.88 MBytes       
[  5]   5.00-6.00   sec   514 MBytes  4.31 Gbits/sec    2   2.10 MBytes 


I know that I need to run iperf through opnsense to test routing performance, not testing how fast opnsense can run the iperf server( or client ) itself but I am failing to understand why opnsense itself can iperf out at 10G  ( either on ax1 or ax0)  but anything that goes routed through it gets capped at 4G... it seams to big of a performance lost added by routing...

P.S - on any scenario if i run more than 1 thread on iperf i can always get 10G but we run some applications that use a single connection and i would like to understand the reason of the above scenario.

P.S 2 - I don't have IDS or any other filtering mechanism enabled, as i don't have any VLAN's configuration. opnsense is operating in a flat network just doing routing.

[svengru]

What you are seeing is pretty much expected when running iperf with a single stream/connection.

Can you run some tests using parallel streams? The -P option is what you need to add.
E.g. -P 10 would be 10 connections.

I still do see a performance issue with 24.7 that is not in 21.4 when it comes to 10gig performance but let's see what results you get with a more realistic test setup.