WireGuard between home OPNsense and FreeBSD VPS: strong iperf3 asymmetry / many

[Jan_S]

Hi everyone,

I am having a performance issue with a WireGuard tunnel between my home OPNsense firewall and a VPS hosted at HostBrr.

Setup:

• Home side: OPNsense as WireGuard server
• Remote side: FreeBSD VPS at HostBrr as WireGuard client
• WireGuard tunnel network: 10.10.10.0/24
• VPS WireGuard IP: 10.10.10.2
• Home LAN: 192.168.3.0/24
• iperf3 target inside the home LAN: 192.168.3.2
• WireGuard MTU is currently set to 1420, according to the guide I followed

The WireGuard tunnel itself is up and working. Routing also works; the VPS can reach the LAN host 192.168.3.2.

Issue:

The performance is highly asymmetric.

From the VPS to the LAN host, I get around 160 Mbit/s:

iperf3 -c 192.168.3.2 -P 4

[SUM]   0.00-10.05  sec   205 MBytes   171 Mbits/sec  305 sender
[SUM]   0.00-10.05  sec   196 MBytes   163 Mbits/sec      receiver

In the reverse direction, using -R, I only get around 16 Mbit/s:

iperf3 -c 192.168.3.2 -P 4 -R

Reverse mode, remote host 192.168.3.2 is sending

[SUM]   0.00-10.02  sec  23.2 MBytes  19.5 Mbits/sec  2828 sender
[SUM]   0.00-10.01  sec  19.4 MBytes  16.2 Mbits/sec       receiver

The very high retransmit count stands out:

Retr: 2828

There are also several intervals in the reverse test showing 0.00 Bytes, so TCP seems to stall completely for short periods.

What I suspect:

I am not sure whether this is caused by one of the following:

• MTU/MSS issue in the WireGuard tunnel, even though MTU is currently 1420
• Missing or incorrect TCP MSS clamping on OPNsense
• Upload limit or packet loss on the home connection
• VPS/provider issue at HostBRR
• Firewall/NAT rule issue on OPNsense
• CPU limitation on either OPNsense or the VPS

Questions:

• Is 1420 a reasonable MTU for this setup, or should I still test lower values such as 1380 or 1360?
• Should I enable TCP MSS clamping on OPNsense? If yes, on which interface/rule and with which value?
• Where should I check on OPNsense for packet drops, state issues, or packet loss?
• Are there any known issues or best practices when using OPNsense as the WireGuard server and FreeBSD as the client?
• Could this kind of asymmetry be caused by the home upload link, even though the retransmit count is so high?

Any hints on what I should check on the OPNsense side would be appreciated.

Thanks!

[Jan_S]

Additional Speedtest Results

To rule out a general bandwidth limitation on either side, I also ran Ookla Speedtest CLI directly on both endpoints.

FreeBSD VPS at HostBRR:

Speedtest by Ookla

Server: FynnCloud - Kassel (id: 62165)
ISP: GHOSTnet
Idle Latency: 0.79 ms (jitter: 0.03 ms)
Download: 4751.85 Mbps
Upload:  5532.22 Mbps
Packet Loss: Not available


PC at home behind OPNsense:

Speedtest by Ookla

Server: Init7 AG - Dielsdorf (id: 70609)
ISP: Init7
Idle Latency: 0.35 ms (jitter: 0.01 ms)
Download: 7373.09 Mbps
Upload:  23203.52 Mbps
Packet Loss: 0.0%


So both sides have plenty of available bandwidth outside the WireGuard tunnel. The issue seems to be specific to the WireGuard path, routing, MTU/MSS, firewall/state handling, or possibly the interaction between OPNsense and the FreeBSD VPS.