Show Posts
1
22.7 Legacy Series / Getting 25gbits throughput virtualized
« on: December 03, 2022, 11:09:58 pm »
I am seeing some odd effects running opnsense virtualized on proxmox depending on what settings I have for CRC, TSO and LRO. When I have them disabled I get the expected speed from PC to my ISP (10gbits), however the speed to the router itself is "slow" in one direction. If I enable them I get equal speed for up/down to the router and from the router to the ISP I get the more or less max expected, however PC to ISP is then running in the low mbits.
I have the following setup:
opnsense (22.7.9) as KVM on proxmox (7.2-7) with the following HW config:
- 1 socket, 8 cores
- 8 GB memory
- WAN - SFP28, (25gbits) virtio driver, multiqueue 8
- LAN - SFP+, (10gbits) virtio driver, multiqueue 4
- 10GB disk
My tests:
CRC, TSO, LRO all disabled - Linux PC (LAN) <-> Opnsense
CRC, TSO, LRO all disabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)
CRC, TSO, LRO all disabled - Opnsense (SSH) <-> ISP
CRC, TSO, LRO all enabled - Linux PC (LAN) <-> Opnsense
CRC, TSO, LRO all enabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)
CRC, TSO, LRO all enabled - Opnsense (SSH) <-> ISP
What is the correct settings for CRC, TSO, LRO or any other settings when you are virtualizing opnsense? Offloading seems to bring the most performance but breaks traffic going LAN <-> WAN.
I have the following setup:
opnsense (22.7.9) as KVM on proxmox (7.2-7) with the following HW config:
- 1 socket, 8 cores
- 8 GB memory
- WAN - SFP28, (25gbits) virtio driver, multiqueue 8
- LAN - SFP+, (10gbits) virtio driver, multiqueue 4
- 10GB disk
My tests:
CRC, TSO, LRO all disabled - Linux PC (LAN) <-> Opnsense
Code: [Select]
└>iperf3 -c 10.10.10.1
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.29 GBytes 5.40 Gbits/sec 364 sender
[ 5] 0.00-10.00 sec 6.28 GBytes 5.39 Gbits/sec receiver
Code: [Select]
└>iperf3 -c 10.10.10.1 -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.90 GBytes 8.51 Gbits/sec 4460 sender
[ 5] 0.00-10.00 sec 9.90 GBytes 8.51 Gbits/sec receiver
CRC, TSO, LRO all disabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)
Code: [Select]
└>iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.3 GBytes 8.83 Gbits/sec 1560 sender
[ 5] 0.00-10.04 sec 10.3 GBytes 8.78 Gbits/sec receiver
Code: [Select]
└>iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 9.41 GBytes 8.05 Gbits/sec 12735 sender
[ 5] 0.00-10.00 sec 9.40 GBytes 8.08 Gbits/sec receiver
Code: [Select]
└>speedtest
Speedtest by Ookla
Server: Init7 AG - Winterthur (id: 43030)
ISP: Init7 (Switzerland) Ltd.
Idle Latency: 0.52 ms (jitter: 0.05ms, low: 0.48ms, high: 0.55ms)
Download: 9041.70 Mbps (data used: 4.5 GB)
3.43 ms (jitter: 12.13ms, low: 0.54ms, high: 425.29ms)
Upload: 8602.60 Mbps (data used: 5.8 GB)
0.90 ms (jitter: 0.82ms, low: 0.39ms, high: 5.46ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/c48584ab-f123-4648-85c8-952268bc00fb
CRC, TSO, LRO all disabled - Opnsense (SSH) <-> ISP
Code: [Select]
root@ch2:~ # iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.3 GBytes 9.66 Gbits/sec 469 sender
[ 5] 0.00-10.00 sec 11.3 GBytes 9.66 Gbits/sec receiver
Code: [Select]
root@ch2:~ # iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 7.10 GBytes 6.10 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 7.10 GBytes 6.10 Gbits/sec receiver
Code: [Select]
root@ch2:~ # speedtest
Speedtest by Ookla
Server: Init7 AG - Winterthur (id: 43030)
ISP: Init7 (Switzerland) Ltd.
Idle Latency: 0.55 ms (jitter: 0.04ms, low: 0.49ms, high: 0.60ms)
Download: 8544.89 Mbps (data used: 4.5 GB)
3.63 ms (jitter: 8.25ms, low: 0.46ms, high: 222.99ms)
Upload: 15508.16 Mbps (data used: 18.9 GB)
4.48 ms (jitter: 1.59ms, low: 0.38ms, high: 15.79ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/8af4c59b-8437-4438-9871-17dea9e03a01
CRC, TSO, LRO all enabled - Linux PC (LAN) <-> Opnsense
Code: [Select]
└>iperf3 -c 10.10.10.1
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 121 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiver
Code: [Select]
└>iperf3 -c 10.10.10.1 -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.98 GBytes 8.57 Gbits/sec 2433 sender
[ 5] 0.00-10.00 sec 9.98 GBytes 8.57 Gbits/sec receiver
CRC, TSO, LRO all enabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)
Code: [Select]
└>iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.07 MBytes 3.42 Mbits/sec 1082 sender
[ 5] 0.00-10.04 sec 3.71 MBytes 3.10 Mbits/sec receiver
Code: [Select]
└>iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 426 KBytes 347 Kbits/sec 359 sender
[ 5] 0.00-10.00 sec 332 KBytes 272 Kbits/sec receiver
Code: [Select]
└>speedtest
Speedtest by Ookla
Server: Init7 AG - Winterthur (id: 43030)
ISP: Init7 (Switzerland) Ltd.
Idle Latency: 0.60 ms (jitter: 0.05ms, low: 0.57ms, high: 0.68ms)
Download: 1.25 Mbps (data used: 2.2 MB)
0.78 ms (jitter: 0.14ms, low: 0.54ms, high: 1.91ms)
Upload: 14.19 Mbps (data used: 22.6 MB)
0.72 ms (jitter: 0.11ms, low: 0.55ms, high: 4.52ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/4de393c4-a8ea-4275-8841-102a3351bbdb
CRC, TSO, LRO all enabled - Opnsense (SSH) <-> ISP
Code: [Select]
root@ch2:~ # iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.2 GBytes 9.58 Gbits/sec 391 sender
[ 5] 0.00-10.00 sec 11.1 GBytes 9.58 Gbits/sec receiver
Code: [Select]
root@ch2:~ # iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 22.5 GBytes 19.3 Gbits/sec 55592 sender
[ 5] 0.00-10.00 sec 22.5 GBytes 19.3 Gbits/sec receiver
Code: [Select]
root@ch2:~ # speedtest
Speedtest by Ookla
Server: Init7 AG - Winterthur (id: 43030)
ISP: Init7 (Switzerland) Ltd.
Idle Latency: 0.47 ms (jitter: 0.03ms, low: 0.42ms, high: 0.50ms)
Download: 19210.94 Mbps (data used: 9.9 GB)
1.91 ms (jitter: 2.77ms, low: 0.42ms, high: 26.38ms)
Upload: 16331.65 Mbps (data used: 14.3 GB)
0.54 ms (jitter: 0.40ms, low: 0.36ms, high: 5.90ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/3d590cab-5a1b-488e-94cb-d34db70262f5
What is the correct settings for CRC, TSO, LRO or any other settings when you are virtualizing opnsense? Offloading seems to bring the most performance but breaks traffic going LAN <-> WAN.