wireguard slow download

[mnaim]

Hi,

I have following setup:
Public VM(Server1), Opnsense last version, 400/400 internet connection, Wireguard kmod, NAT from wireguard to WAN.

To Server1 I have connected client (Server2-public VM-Ubuntu) via Wireguard to access internet only via wireguard tunnel (0.0.0.0/0). Server 2 connectivity is 400/400 too.

Both servers are Xeon based (enough performance to encrypt/decrypt), when active on full speed 40% of one core is used.

On both servers running iperf 3 or download from test server "wget https://speed.hetzner.de/10GB.bin" I have full internet connectivity.

When wireguard is active Server2 upload is near full speed(5-10% drop) but download drop is to 2-3MBytes/sec = 16-24MBit/sec.

I start tunning everything to narrow problem for 5days and now Im lost.

I think problem is MTU. Both server WAN is 1500. On both WG 1412 is set as MTU.

Playing with MSS on Server1 on WAN or WG interface - no impact.

Attached is packet capture. My public ip is redacted, 88.198.248.254=speed.hetzner.de, 192.168.4.14=ip wg server2 as described in attached diagram.

diagram


capture_vtnet0-server1-wan


capture_wg0-server1-wg



Packet capture shows that size of TCP packet from test server is 1426, but on first line MSS is sent 1372 (which is correct 1412-40).

After while 662 packets later speed decreases and some black lines appeared, later speed stabilize around 16-24Mbit/sec, which is far too low that upload around 360-380Mbit with WG and 400Mbit download without WG.

Interesting point, that download is ok for UDP (iperf3), but low for TCP (iperf3).

Now Im really out of options what to try next to get download speed back.

Thanks