Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - sschueller

Pages1
#1
22.7 Legacy Series / Getting 25gbits throughput virtualized
December 03, 2022, 11:09:58 PM
I am seeing some odd effects running opnsense virtualized on proxmox depending on what settings I have for CRC, TSO and LRO. When I have them disabled I get the expected speed from PC to my ISP (10gbits), however the speed to the router itself is "slow" in one direction. If I enable them I get equal speed for up/down to the router and from the router to the ISP I get the more or less max expected, however PC to ISP is then running in the low mbits.

I have the following setup:

opnsense (22.7.9) as KVM on proxmox (7.2-7) with the following HW config:
- 1 socket, 8 cores
- 8 GB memory
- WAN - SFP28, (25gbits) virtio driver, multiqueue 8
- LAN - SFP+, (10gbits) virtio driver, multiqueue 4
- 10GB disk

My tests:

CRC, TSO, LRO all disabled - Linux PC (LAN) <-> Opnsense

Code Select

└>iperf3 -c 10.10.10.1   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.29 GBytes  5.40 Gbits/sec  364             sender
[  5]   0.00-10.00  sec  6.28 GBytes  5.39 Gbits/sec                  receiver

Code Select

└>iperf3 -c 10.10.10.1 -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.90 GBytes  8.51 Gbits/sec  4460             sender
[  5]   0.00-10.00  sec  9.90 GBytes  8.51 Gbits/sec                  receiver



CRC, TSO, LRO all disabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)

Code Select

└>iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.3 GBytes  8.83 Gbits/sec  1560             sender
[  5]   0.00-10.04  sec  10.3 GBytes  8.78 Gbits/sec                  receiver

Code Select

└>iperf3 -c speedtest.init7.net -R               
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  9.41 GBytes  8.05 Gbits/sec  12735             sender
[  5]   0.00-10.00  sec  9.40 GBytes  8.08 Gbits/sec                  receiver


Code Select

└>speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.52 ms   (jitter: 0.05ms, low: 0.48ms, high: 0.55ms)
    Download:  9041.70 Mbps (data used: 4.5 GB)                                                   
                  3.43 ms   (jitter: 12.13ms, low: 0.54ms, high: 425.29ms)
      Upload:  8602.60 Mbps (data used: 5.8 GB)                                                   
                  0.90 ms   (jitter: 0.82ms, low: 0.39ms, high: 5.46ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/c48584ab-f123-4648-85c8-952268bc00fb



CRC, TSO, LRO all disabled - Opnsense (SSH) <-> ISP

Code Select

root@ch2:~ # iperf3 -c speedtest.init7.net 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.3 GBytes  9.66 Gbits/sec  469             sender
[  5]   0.00-10.00  sec  11.3 GBytes  9.66 Gbits/sec                  receiver

Code Select

root@ch2:~ # iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.10 GBytes  6.10 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  7.10 GBytes  6.10 Gbits/sec                  receiver


Code Select

root@ch2:~ # speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.55 ms   (jitter: 0.04ms, low: 0.49ms, high: 0.60ms)
    Download:  8544.89 Mbps (data used: 4.5 GB)                                                   
                  3.63 ms   (jitter: 8.25ms, low: 0.46ms, high: 222.99ms)
      Upload: 15508.16 Mbps (data used: 18.9 GB)                                                   
                  4.48 ms   (jitter: 1.59ms, low: 0.38ms, high: 15.79ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/8af4c59b-8437-4438-9871-17dea9e03a01


CRC, TSO, LRO all enabled - Linux PC (LAN) <-> Opnsense

Code Select

└>iperf3 -c 10.10.10.1   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec  121             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver


Code Select

└>iperf3 -c 10.10.10.1 -R           
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.98 GBytes  8.57 Gbits/sec  2433             sender
[  5]   0.00-10.00  sec  9.98 GBytes  8.57 Gbits/sec                  receiver


CRC, TSO, LRO all enabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)

Code Select

└>iperf3 -c speedtest.init7.net   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.07 MBytes  3.42 Mbits/sec  1082             sender
[  5]   0.00-10.04  sec  3.71 MBytes  3.10 Mbits/sec                  receiver


Code Select

└>iperf3 -c speedtest.init7.net -R             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   426 KBytes   347 Kbits/sec  359             sender
[  5]   0.00-10.00  sec   332 KBytes   272 Kbits/sec                  receiver


Code Select

└>speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.60 ms   (jitter: 0.05ms, low: 0.57ms, high: 0.68ms)
    Download:     1.25 Mbps (data used: 2.2 MB)                                                   
                  0.78 ms   (jitter: 0.14ms, low: 0.54ms, high: 1.91ms)
      Upload:    14.19 Mbps (data used: 22.6 MB)                                                   
                  0.72 ms   (jitter: 0.11ms, low: 0.55ms, high: 4.52ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/4de393c4-a8ea-4275-8841-102a3351bbdb


CRC, TSO, LRO all enabled - Opnsense (SSH) <-> ISP

Code Select

root@ch2:~ # iperf3 -c speedtest.init7.net     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.2 GBytes  9.58 Gbits/sec  391             sender
[  5]   0.00-10.00  sec  11.1 GBytes  9.58 Gbits/sec                  receiver

Code Select

root@ch2:~ # iperf3 -c speedtest.init7.net -R             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  22.5 GBytes  19.3 Gbits/sec  55592             sender
[  5]   0.00-10.00  sec  22.5 GBytes  19.3 Gbits/sec                  receiver

Code Select

root@ch2:~ # speedtest

   Speedtest by Ookla

      Server: Init7 AG - Winterthur (id: 43030)
         ISP: Init7 (Switzerland) Ltd.
Idle Latency:     0.47 ms   (jitter: 0.03ms, low: 0.42ms, high: 0.50ms)
    Download: 19210.94 Mbps (data used: 9.9 GB)                                                   
                  1.91 ms   (jitter: 2.77ms, low: 0.42ms, high: 26.38ms)
      Upload: 16331.65 Mbps (data used: 14.3 GB)                                                   
                  0.54 ms   (jitter: 0.40ms, low: 0.36ms, high: 5.90ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/3d590cab-5a1b-488e-94cb-d34db70262f5


What is the correct settings for CRC, TSO, LRO or any other settings when you are virtualizing opnsense? Offloading seems to bring the most performance but breaks traffic going LAN <-> WAN.

#2
22.1 Legacy Series / Re: High Packet Loss/CRC errors for Tx on axgbe (DEC2750/OPNsense 22.1)
November 27, 2022, 07:03:44 PM
Were you able to solve this issue?

I appear to be hitting the same problem.

I have a proxmox host with opnsense running as a guest. When I have CRC, TSO and LRO disabled I hit bottlenecks (~5-6gbit in one direction and ~7-8gbit in the other). When I enable CRC,TSO and LRO I am able to hit ~9gbit from host to router and ~23gbit from router to ISP in both directions however going across the interfaces (NAT) I get pushed down to ~1mbit.
Pages1