Getting 25gbits throughput virtualized

[sschueller]

I am seeing some odd effects running opnsense virtualized on proxmox depending on what settings I have for CRC, TSO and LRO. When I have them disabled I get the expected speed from PC to my ISP (10gbits), however the speed to the router itself is "slow" in one direction. If I enable them I get equal speed for up/down to the router and from the router to the ISP I get the more or less max expected, however PC to ISP is then running in the low mbits.

I have the following setup:

opnsense (22.7.9) as KVM on proxmox (7.2-7) with the following HW config:
- 1 socket, 8 cores
- 8 GB memory
- WAN - SFP28, (25gbits) virtio driver, multiqueue 8
- LAN - SFP+, (10gbits) virtio driver, multiqueue 4
- 10GB disk

My tests:

CRC, TSO, LRO all disabled - Linux PC (LAN) <-> Opnsense

Code: [Select]

└>iperf3 -c 10.10.10.1   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.29 GBytes  5.40 Gbits/sec  364             sender
[  5]   0.00-10.00  sec  6.28 GBytes  5.39 Gbits/sec                  receiver


Code: [Select]

└>iperf3 -c 10.10.10.1 -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.90 GBytes  8.51 Gbits/sec  4460             sender
[  5]   0.00-10.00  sec  9.90 GBytes  8.51 Gbits/sec                  receiver


CRC, TSO, LRO all disabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)

Code: [Select]

└>iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.3 GBytes  8.83 Gbits/sec  1560             sender
[  5]   0.00-10.04  sec  10.3 GBytes  8.78 Gbits/sec                  receiver


Code: [Select]

└>iperf3 -c speedtest.init7.net -R               
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  9.41 GBytes  8.05 Gbits/sec  12735             sender
[  5]   0.00-10.00  sec  9.40 GBytes  8.08 Gbits/sec                  receiver


Code: [Select]

└>speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.52 ms   (jitter: 0.05ms, low: 0.48ms, high: 0.55ms)
    Download:  9041.70 Mbps (data used: 4.5 GB)                                                   
                  3.43 ms   (jitter: 12.13ms, low: 0.54ms, high: 425.29ms)
      Upload:  8602.60 Mbps (data used: 5.8 GB)                                                   
                  0.90 ms   (jitter: 0.82ms, low: 0.39ms, high: 5.46ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/c48584ab-f123-4648-85c8-952268bc00fb


CRC, TSO, LRO all disabled - Opnsense (SSH) <-> ISP

Code: [Select]

root@ch2:~ # iperf3 -c speedtest.init7.net 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.3 GBytes  9.66 Gbits/sec  469             sender
[  5]   0.00-10.00  sec  11.3 GBytes  9.66 Gbits/sec                  receiver


Code: [Select]

root@ch2:~ # iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.10 GBytes  6.10 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  7.10 GBytes  6.10 Gbits/sec                  receiver


Code: [Select]

root@ch2:~ # speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.55 ms   (jitter: 0.04ms, low: 0.49ms, high: 0.60ms)
    Download:  8544.89 Mbps (data used: 4.5 GB)                                                   
                  3.63 ms   (jitter: 8.25ms, low: 0.46ms, high: 222.99ms)
      Upload: 15508.16 Mbps (data used: 18.9 GB)                                                   
                  4.48 ms   (jitter: 1.59ms, low: 0.38ms, high: 15.79ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/8af4c59b-8437-4438-9871-17dea9e03a01

CRC, TSO, LRO all enabled - Linux PC (LAN) <-> Opnsense

Code: [Select]

└>iperf3 -c 10.10.10.1   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec  121             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver


Code: [Select]

└>iperf3 -c 10.10.10.1 -R           
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.98 GBytes  8.57 Gbits/sec  2433             sender
[  5]   0.00-10.00  sec  9.98 GBytes  8.57 Gbits/sec                  receiver

CRC, TSO, LRO all enabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)

Code: [Select]

└>iperf3 -c speedtest.init7.net   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.07 MBytes  3.42 Mbits/sec  1082             sender
[  5]   0.00-10.04  sec  3.71 MBytes  3.10 Mbits/sec                  receiver


Code: [Select]

└>iperf3 -c speedtest.init7.net -R             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   426 KBytes   347 Kbits/sec  359             sender
[  5]   0.00-10.00  sec   332 KBytes   272 Kbits/sec                  receiver


Code: [Select]

└>speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.60 ms   (jitter: 0.05ms, low: 0.57ms, high: 0.68ms)
    Download:     1.25 Mbps (data used: 2.2 MB)                                                   
                  0.78 ms   (jitter: 0.14ms, low: 0.54ms, high: 1.91ms)
      Upload:    14.19 Mbps (data used: 22.6 MB)                                                   
                  0.72 ms   (jitter: 0.11ms, low: 0.55ms, high: 4.52ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/4de393c4-a8ea-4275-8841-102a3351bbdb

CRC, TSO, LRO all enabled - Opnsense (SSH) <-> ISP

Code: [Select]

root@ch2:~ # iperf3 -c speedtest.init7.net     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.2 GBytes  9.58 Gbits/sec  391             sender
[  5]   0.00-10.00  sec  11.1 GBytes  9.58 Gbits/sec                  receiver


Code: [Select]

root@ch2:~ # iperf3 -c speedtest.init7.net -R             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  22.5 GBytes  19.3 Gbits/sec  55592             sender
[  5]   0.00-10.00  sec  22.5 GBytes  19.3 Gbits/sec                  receiver


Code: [Select]

root@ch2:~ # speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.47 ms   (jitter: 0.03ms, low: 0.42ms, high: 0.50ms)
    Download: 19210.94 Mbps (data used: 9.9 GB)                                                   
                  1.91 ms   (jitter: 2.77ms, low: 0.42ms, high: 26.38ms)
      Upload: 16331.65 Mbps (data used: 14.3 GB)                                                   
                  0.54 ms   (jitter: 0.40ms, low: 0.36ms, high: 5.90ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/3d590cab-5a1b-488e-94cb-d34db70262f5

What is the correct settings for CRC, TSO, LRO or any other settings when you are virtualizing opnsense? Offloading seems to bring the most performance but breaks traffic going LAN <-> WAN.

[lilsense]

why virtualized?

Let's presume this is just a test, and nothing is running on this device and all the services are stopped and you are only attempting to do is to use iperf3 as a test. I am not sure you have enough power for 25Gbps interface. you may tick for half sec, but I even doubt you'd hit it.

[lewald]

Maybee this help

https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/