Wireguard Performance Capped - why?

[Gizmo]

Hi all,

Looking for some advice on further tuning ideas to maxmise my Wireguard (Via Nord VPN) performance.

This time totally stalled at how to get my Wireguard VPN performance close to my 1Gb internet connection speed. Currently caps out around 450 to 550Mbps. The speed completely flatlines which leads me to believe its simply a setting which is maxing the throughput/processing.

Firstly, my ISP allows these speeds and have done direct connection to internet router getting about 975Mbps.


Key Questions I have

• Does the DNS config affect speed? (Currently using Unbound in forwarding mode to Quad9 Servers)
• Are there specific turnables settings others have used and found a speed boost?
• What specific MSS and MTU settings were used and where did you apply these?

I have played around with the MTU and MSS settings, between 1380 to 1420. Not seen any major jump across a range of combinations. Additionally not sure where is the best place to enter these as there seems to be several locations to do it

• The wireguard tunnel
• WG interface
• LAN interface
• Interface normalisation settings
• System settings

Use Case
Simple home setup using Nord VPN for wireguard, just trying to get maximum speed.


Current Setup

• Protectli FW6Br2 Intel i3-8130U 2.2Ghz 2 core 4 thread CPU with 16GB DDR4 Ram and 256GB SSD (According to Protectli Wireguard speeds of 900Mbps capable)
• OPNsense 23.7.1_3-amd64
• FreeBSD 13.2-RELEASE-p2
• OpenSSL 1.1.1v 1 Aug 2023

Test ResultsTesting via ethernet cable into LAN port via Speednet CLI Test

Speedtest by Ookla

      Server: Network Solutions Group - Sydney (id: 30430)
         ISP: GSL Networks Pty
Idle Latency:    12.25 ms   (jitter: 4.04ms, low: 8.57ms, high: 16.21ms)
    Download:   455.10 Mbps [==========-         ] 54%   - latency: 273.32 ms       Download:   464.39 Mbps [===========\        ] 55%   - latency: 273.32 ms       Download:   465.29 Mbps [===========|        ] 56%   - latency: 273.32 ms       Download:   465.38 Mbps [===========/        ] 56%   - latency: 273.32 ms 

Upload:    45.21 Mbps (data used: 35.0 MB)                                                   

                 47.27 ms   (jitter: 4.05ms, low: 14.23ms, high: 81.45ms)

Opnsense Setup
LAN Interface MTU = 1420
WG Interface MTU &  MSS = 1420
Using Unbound DNS forwarding to Cloud9 servers - Not using local resolver - Unsure which is best for my application

Notable Turnables I've adjusted based on various gudes - In particular https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/
https://forum.opnsense.org/index.php?topic=24409.msg116941#msg116941

kern.ipc.maxsockbuf = 614400000
net.inet.rss.bits = 2
net.inet.rss.enabled = 1
net.inet.tcp.abc_1_var = 52
net.inet.tcp.minmss = 536
net.inet.tcp.mssdflt = 1240
net.inet.udp.checksum = 1
net.inet.udp.maxdgram = 57344
net.isr.defaultqlimit = 2048
net.isr.dispatch = deferred
net.isr.maxthreads = -1
net.local.dgram.maxdgram = 8192
net.pf.source_nodes_hashsize = 1048576
set.hw.ibrs_disable = 1
vfs.read_max = 32


Any help or advice much appreciated.

[CJ]

I don't have any suggestions for you as I use WG for a road warrior setup, but it does remind me to try some performance testing on mine.

[newsense]

It's a decent system you have there, probably capped by the CPU for the encryption bits - a quad core would have been better.

What's the CPU % like when you do those tests ?

[Gizmo]

Hi there,

Here is a snap of my CPU usage while carrying out a speedtest CLI via LAN with ethernet cable to firewall.

Spikes to 60% to 80%. Two screen shot CPU charts attached.

Code: [Select]

Speedtest by Ookla

      Server: GSL Networks - Sydney (id: 44735)
         ISP: GSL Networks Pty
Idle Latency:    10.62 ms   (jitter: 1.46ms, low: 8.15ms, high: 13.23ms)
    Download:   524.55 Mbps (data used: 582.9 MB)                                                   
                 28.76 ms   (jitter: 16.32ms, low: 12.80ms, high: 267.36ms)
      Upload:    44.16 Mbps (data used: 48.1 MB)                                                   
                 10.01 ms   (jitter: 8.59ms, low: 5.30ms, high: 363.70ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/41e54725-8ec0-4f0a-a840-d3c2d397bf1d



[Gizmo]

It's a decent system you have there, probably capped by the CPU for the encryption bits - a quad core would have been better.

What's the CPU % like when you do those tests ?

Any ideas? Currently with latest Opnsense and updated Wireguard KMOD, sitting at 500mbps. Surely there is a small tweak on the turntables or similar. I'm still thinking its running on one core.

[gadgetguy]

I'd be interested as well.
My ISP bandwidth is 300 Mbps up/down. 
With 23.1 WG was about 10% off that.
With 23.7 it is about 40% off that with no changes on my side. 

[bbchucks]

Have the same issue, if i bypass the opnsense and use wireguard on my macbook i get about 800Mbps to 900, close to 1Gigabit.
However when i use wireguard on the opnsense box (HP T720) then my speeds drop down to 250-280Mbps.

Please let me know if you found a workaround.

Hi all,

Looking for some advice on further tuning ideas to maxmise my Wireguard (Via Nord VPN) performance.

This time totally stalled at how to get my Wireguard VPN performance close to my 1Gb internet connection speed. Currently caps out around 450 to 550Mbps. The speed completely flatlines which leads me to believe its simply a setting which is maxing the throughput/processing.

Firstly, my ISP allows these speeds and have done direct connection to internet router getting about 975Mbps.


Key Questions I have

• Does the DNS config affect speed? (Currently using Unbound in forwarding mode to Quad9 Servers)
• Are there specific turnables settings others have used and found a speed boost?
• What specific MSS and MTU settings were used and where did you apply these?

I have played around with the MTU and MSS settings, between 1380 to 1420. Not seen any major jump across a range of combinations. Additionally not sure where is the best place to enter these as there seems to be several locations to do it

• The wireguard tunnel
• WG interface
• LAN interface
• Interface normalisation settings
• System settings

Use Case
Simple home setup using Nord VPN for wireguard, just trying to get maximum speed.


Current Setup

• Protectli FW6Br2 Intel i3-8130U 2.2Ghz 2 core 4 thread CPU with 16GB DDR4 Ram and 256GB SSD (According to Protectli Wireguard speeds of 900Mbps capable)
• OPNsense 23.7.1_3-amd64
• FreeBSD 13.2-RELEASE-p2
• OpenSSL 1.1.1v 1 Aug 2023

Test ResultsTesting via ethernet cable into LAN port via Speednet CLI Test

Speedtest by Ookla

      Server: Network Solutions Group - Sydney (id: 30430)
         ISP: GSL Networks Pty
Idle Latency:    12.25 ms   (jitter: 4.04ms, low: 8.57ms, high: 16.21ms)
    Download:   455.10 Mbps [==========-         ] 54%   - latency: 273.32 ms       Download:   464.39 Mbps [===========\        ] 55%   - latency: 273.32 ms       Download:   465.29 Mbps [===========|        ] 56%   - latency: 273.32 ms       Download:   465.38 Mbps [===========/        ] 56%   - latency: 273.32 ms 

Upload:    45.21 Mbps (data used: 35.0 MB)                                                   

                 47.27 ms   (jitter: 4.05ms, low: 14.23ms, high: 81.45ms)

Opnsense Setup
LAN Interface MTU = 1420
WG Interface MTU &  MSS = 1420
Using Unbound DNS forwarding to Cloud9 servers - Not using local resolver - Unsure which is best for my application

Notable Turnables I've adjusted based on various gudes - In particular https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/
https://forum.opnsense.org/index.php?topic=24409.msg116941#msg116941

kern.ipc.maxsockbuf = 614400000
net.inet.rss.bits = 2
net.inet.rss.enabled = 1
net.inet.tcp.abc_1_var = 52
net.inet.tcp.minmss = 536
net.inet.tcp.mssdflt = 1240
net.inet.udp.checksum = 1
net.inet.udp.maxdgram = 57344
net.isr.defaultqlimit = 2048
net.isr.dispatch = deferred
net.isr.maxthreads = -1
net.local.dgram.maxdgram = 8192
net.pf.source_nodes_hashsize = 1048576
set.hw.ibrs_disable = 1
vfs.read_max = 32


Any help or advice much appreciated.

[gadgetguy]

I found I had better performance when I turned off Zenarmor.

[CJ]

I ended up having to do some MTU tuning recently while connecting to WG over a 5G network.  Oddly, I haven't had to do anything for my Android device and it's 5G connection.

I don't recall the exact MTU limit that the 5G network had but I set WG to 1280 because the 5G networks are generally all IPv6.  After that my speeds picked up considerably.

[bedub1]

EDIT:  I've been banned from the forum?  For what, posting the solution?  Welll...no solution for you!

[CJ]

I was running an i5-7200u.  Wireguard was about 600mbps, with cpu at 100%.
I upgraded to a Xeon E3-1285 V4.  Wireguard is about 600mbps, with cpu at 100%.

This new Xeon is nearly 4x faster than the old one.

What do you see under System->Diagnostics->Activity when you are getting 600m on WG?

[bedub1]

EDIT:  I've been banned from the forum?  For what, posting the solution?  Welll...no solution for you!

[bedub1]

EDIT:  I've been banned from the forum?  For what, posting the solution?  Welll...no solution for you!

[bedub1]

EDIT:  I've been banned from the forum?  For what, posting the solution?  Welll...no solution for you! 

[CJ]