I am seeing some odd effects running opnsense virtualized on proxmox depending on what settings I have for CRC, TSO and LRO. When I have them disabled I get the expected speed from PC to my ISP (10gbits), however the speed to the router itself is "slow" in one direction. If I enable them I get equal speed for up/down to the router and from the router to the ISP I get the more or less max expected, however PC to ISP is then running in the low mbits.
I have the following setup:
opnsense (22.7.9) as KVM on proxmox (7.2-7) with the following HW config:
- 1 socket, 8 cores
- 8 GB memory
- WAN - SFP28, (25gbits) virtio driver, multiqueue 8
- LAN - SFP+, (10gbits) virtio driver, multiqueue 4
- 10GB disk
My tests:
CRC, TSO, LRO all disabled - Linux PC (LAN) <-> Opnsense
└>iperf3 -c 10.10.10.1
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.29 GBytes 5.40 Gbits/sec 364 sender
[ 5] 0.00-10.00 sec 6.28 GBytes 5.39 Gbits/sec receiver
└>iperf3 -c 10.10.10.1 -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.90 GBytes 8.51 Gbits/sec 4460 sender
[ 5] 0.00-10.00 sec 9.90 GBytes 8.51 Gbits/sec receiver
CRC, TSO, LRO all disabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)
└>iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.3 GBytes 8.83 Gbits/sec 1560 sender
[ 5] 0.00-10.04 sec 10.3 GBytes 8.78 Gbits/sec receiver
└>iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 9.41 GBytes 8.05 Gbits/sec 12735 sender
[ 5] 0.00-10.00 sec 9.40 GBytes 8.08 Gbits/sec receiver
└>speedtest
Speedtest by Ookla
Server: Init7 AG - Winterthur (id: 43030)
ISP: Init7 (Switzerland) Ltd.
Idle Latency: 0.52 ms (jitter: 0.05ms, low: 0.48ms, high: 0.55ms)
Download: 9041.70 Mbps (data used: 4.5 GB)
3.43 ms (jitter: 12.13ms, low: 0.54ms, high: 425.29ms)
Upload: 8602.60 Mbps (data used: 5.8 GB)
0.90 ms (jitter: 0.82ms, low: 0.39ms, high: 5.46ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/c48584ab-f123-4648-85c8-952268bc00fb
CRC, TSO, LRO all disabled - Opnsense (SSH) <-> ISP
root@ch2:~ # iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.3 GBytes 9.66 Gbits/sec 469 sender
[ 5] 0.00-10.00 sec 11.3 GBytes 9.66 Gbits/sec receiver
root@ch2:~ # iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 7.10 GBytes 6.10 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 7.10 GBytes 6.10 Gbits/sec receiver
root@ch2:~ # speedtest
Speedtest by Ookla
Server: Init7 AG - Winterthur (id: 43030)
ISP: Init7 (Switzerland) Ltd.
Idle Latency: 0.55 ms (jitter: 0.04ms, low: 0.49ms, high: 0.60ms)
Download: 8544.89 Mbps (data used: 4.5 GB)
3.63 ms (jitter: 8.25ms, low: 0.46ms, high: 222.99ms)
Upload: 15508.16 Mbps (data used: 18.9 GB)
4.48 ms (jitter: 1.59ms, low: 0.38ms, high: 15.79ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/8af4c59b-8437-4438-9871-17dea9e03a01
CRC, TSO, LRO all enabled - Linux PC (LAN) <-> Opnsense
└>iperf3 -c 10.10.10.1
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 121 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiver
└>iperf3 -c 10.10.10.1 -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.98 GBytes 8.57 Gbits/sec 2433 sender
[ 5] 0.00-10.00 sec 9.98 GBytes 8.57 Gbits/sec receiver
CRC, TSO, LRO all enabled - Linux PC (LAN) <-> ISP (via Opnsense NAT)
└>iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.07 MBytes 3.42 Mbits/sec 1082 sender
[ 5] 0.00-10.04 sec 3.71 MBytes 3.10 Mbits/sec receiver
└>iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 426 KBytes 347 Kbits/sec 359 sender
[ 5] 0.00-10.00 sec 332 KBytes 272 Kbits/sec receiver
└>speedtest
Speedtest by Ookla
Server: Init7 AG - Winterthur (id: 43030)
ISP: Init7 (Switzerland) Ltd.
Idle Latency: 0.60 ms (jitter: 0.05ms, low: 0.57ms, high: 0.68ms)
Download: 1.25 Mbps (data used: 2.2 MB)
0.78 ms (jitter: 0.14ms, low: 0.54ms, high: 1.91ms)
Upload: 14.19 Mbps (data used: 22.6 MB)
0.72 ms (jitter: 0.11ms, low: 0.55ms, high: 4.52ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/4de393c4-a8ea-4275-8841-102a3351bbdb
CRC, TSO, LRO all enabled - Opnsense (SSH) <-> ISP
root@ch2:~ # iperf3 -c speedtest.init7.net
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.2 GBytes 9.58 Gbits/sec 391 sender
[ 5] 0.00-10.00 sec 11.1 GBytes 9.58 Gbits/sec receiver
root@ch2:~ # iperf3 -c speedtest.init7.net -R
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 22.5 GBytes 19.3 Gbits/sec 55592 sender
[ 5] 0.00-10.00 sec 22.5 GBytes 19.3 Gbits/sec receiver
root@ch2:~ # speedtest
Speedtest by Ookla
Server: Init7 AG - Winterthur (id: 43030)
ISP: Init7 (Switzerland) Ltd.
Idle Latency: 0.47 ms (jitter: 0.03ms, low: 0.42ms, high: 0.50ms)
Download: 19210.94 Mbps (data used: 9.9 GB)
1.91 ms (jitter: 2.77ms, low: 0.42ms, high: 26.38ms)
Upload: 16331.65 Mbps (data used: 14.3 GB)
0.54 ms (jitter: 0.40ms, low: 0.36ms, high: 5.90ms)
Packet Loss: 0.0%
Result URL: https://www.speedtest.net/result/c/3d590cab-5a1b-488e-94cb-d34db70262f5
What is the correct settings for CRC, TSO, LRO or any other settings when you are virtualizing opnsense? Offloading seems to bring the most performance but breaks traffic going LAN <-> WAN.
why virtualized?
Let's presume this is just a test, and nothing is running on this device and all the services are stopped and you are only attempting to do is to use iperf3 as a test. I am not sure you have enough power for 25Gbps interface. you may tick for half sec, but I even doubt you'd hit it.
Maybee this help
https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/