Getting 25gbits throughput virtualized

Started by sschueller, December 03, 2022, 11:09:58 PM

Previous topic - Next topic
Print
Go Down Pages1
User actions
December 03, 2022, 11:09:58 PM
I am seeing some odd effects running opnsense virtualized on proxmox depending on what settings I have for CRC, TSO and LRO. When I have them disabled I get the expected speed from PC to my ISP (10gbits), however the speed to the router itself is "slow" in one direction. If I enable them I get equal speed for up/down to the router and from the router to the ISP I get the more or less max expected, however PC to ISP is then running in the low mbits.

I have the following setup:

opnsense (22.7.9) as KVM on proxmox (7.2-7) with the following HW config:
- 1 socket, 8 cores
- 8 GB memory
- WAN - SFP28, (25gbits) virtio driver, multiqueue 8
- LAN - SFP+, (10gbits) virtio driver, multiqueue 4
- 10GB disk

My tests:

CRC, TSO, LRO all disabled - Linux PC (LAN) <-> Opnsense

Code Select

└>iperf3 -c 10.10.10.1   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.29 GBytes  5.40 Gbits/sec  364             sender
[  5]   0.00-10.00  sec  6.28 GBytes  5.39 Gbits/sec                  receiver

Code Select

└>iperf3 -c 10.10.10.1 -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.90 GBytes  8.51 Gbits/sec  4460             sender
[  5]   0.00-10.00  sec  9.90 GBytes  8.51 Gbits/sec                  receiver



CRC, TSO, LRO all disabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)

Code Select

└>iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.3 GBytes  8.83 Gbits/sec  1560             sender
[  5]   0.00-10.04  sec  10.3 GBytes  8.78 Gbits/sec                  receiver

Code Select

└>iperf3 -c speedtest.init7.net -R               
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  9.41 GBytes  8.05 Gbits/sec  12735             sender
[  5]   0.00-10.00  sec  9.40 GBytes  8.08 Gbits/sec                  receiver


Code Select

└>speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.52 ms   (jitter: 0.05ms, low: 0.48ms, high: 0.55ms)
    Download:  9041.70 Mbps (data used: 4.5 GB)                                                   
                  3.43 ms   (jitter: 12.13ms, low: 0.54ms, high: 425.29ms)
      Upload:  8602.60 Mbps (data used: 5.8 GB)                                                   
                  0.90 ms   (jitter: 0.82ms, low: 0.39ms, high: 5.46ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/c48584ab-f123-4648-85c8-952268bc00fb



CRC, TSO, LRO all disabled - Opnsense (SSH) <-> ISP

Code Select

root@ch2:~ # iperf3 -c speedtest.init7.net 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.3 GBytes  9.66 Gbits/sec  469             sender
[  5]   0.00-10.00  sec  11.3 GBytes  9.66 Gbits/sec                  receiver

Code Select

root@ch2:~ # iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.10 GBytes  6.10 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  7.10 GBytes  6.10 Gbits/sec                  receiver


Code Select

root@ch2:~ # speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.55 ms   (jitter: 0.04ms, low: 0.49ms, high: 0.60ms)
    Download:  8544.89 Mbps (data used: 4.5 GB)                                                   
                  3.63 ms   (jitter: 8.25ms, low: 0.46ms, high: 222.99ms)
      Upload: 15508.16 Mbps (data used: 18.9 GB)                                                   
                  4.48 ms   (jitter: 1.59ms, low: 0.38ms, high: 15.79ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/8af4c59b-8437-4438-9871-17dea9e03a01


CRC, TSO, LRO all enabled - Linux PC (LAN) <-> Opnsense

Code Select

└>iperf3 -c 10.10.10.1   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec  121             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver


Code Select

└>iperf3 -c 10.10.10.1 -R           
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.98 GBytes  8.57 Gbits/sec  2433             sender
[  5]   0.00-10.00  sec  9.98 GBytes  8.57 Gbits/sec                  receiver


CRC, TSO, LRO all enabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)

Code Select

└>iperf3 -c speedtest.init7.net   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.07 MBytes  3.42 Mbits/sec  1082             sender
[  5]   0.00-10.04  sec  3.71 MBytes  3.10 Mbits/sec                  receiver


Code Select

└>iperf3 -c speedtest.init7.net -R             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   426 KBytes   347 Kbits/sec  359             sender
[  5]   0.00-10.00  sec   332 KBytes   272 Kbits/sec                  receiver


Code Select

└>speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.60 ms   (jitter: 0.05ms, low: 0.57ms, high: 0.68ms)
    Download:     1.25 Mbps (data used: 2.2 MB)                                                   
                  0.78 ms   (jitter: 0.14ms, low: 0.54ms, high: 1.91ms)
      Upload:    14.19 Mbps (data used: 22.6 MB)                                                   
                  0.72 ms   (jitter: 0.11ms, low: 0.55ms, high: 4.52ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/4de393c4-a8ea-4275-8841-102a3351bbdb


CRC, TSO, LRO all enabled - Opnsense (SSH) <-> ISP

Code Select

root@ch2:~ # iperf3 -c speedtest.init7.net     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.2 GBytes  9.58 Gbits/sec  391             sender
[  5]   0.00-10.00  sec  11.1 GBytes  9.58 Gbits/sec                  receiver

Code Select

root@ch2:~ # iperf3 -c speedtest.init7.net -R             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  22.5 GBytes  19.3 Gbits/sec  55592             sender
[  5]   0.00-10.00  sec  22.5 GBytes  19.3 Gbits/sec                  receiver

Code Select

root@ch2:~ # speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.47 ms   (jitter: 0.03ms, low: 0.42ms, high: 0.50ms)
    Download: 19210.94 Mbps (data used: 9.9 GB)                                                   
                  1.91 ms   (jitter: 2.77ms, low: 0.42ms, high: 26.38ms)
      Upload: 16331.65 Mbps (data used: 14.3 GB)                                                   
                  0.54 ms   (jitter: 0.40ms, low: 0.36ms, high: 5.90ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/3d590cab-5a1b-488e-94cb-d34db70262f5


What is the correct settings for CRC, TSO, LRO or any other settings when you are virtualizing opnsense? Offloading seems to bring the most performance but breaks traffic going LAN <-> WAN.

December 03, 2022, 11:28:51 PM #1
why virtualized?

Let's presume this is just a test, and nothing is running on this device and all the services are stopped and you are only attempting to do is to use iperf3 as a test. I am not sure you have enough power for 25Gbps interface. you may tick for half sec, but I even doubt you'd hit it.
December 04, 2022, 01:01:50 PM #2
Print
Go Up Pages1
User actions