Wireguard site to site file transfer issue

[ditch9]

I have two sites. Site A and site B.

On site A OPNsense is the edge device. It has a DHCP WAN cable connection and MTU ping testing has verified my MTU settings of 1420 and MSS settings of 1380 are what I need. The WAN speed is 1gbit down 300mbits up. I have multiple remote access Wireguard tunnels in addition to the site to site on this device and there are no issues.

On site B OPNsense sites behind a DD WRT router which is the edge device. DD WRT has a DHCP WAN fiber connection which has the same MTU/MSS as site A. The WAN speed is 300mbit both ways. OPNsense in visualized in Proxmox on this site.

My issue is that file transfers using, wget, curl, rsync, and scp are very slow. Less than 1 megabyte a second. If I run rsync with -ahP the speed will be 100MB/s + for a short time then drop to sub 500kB/s and stay there. This happens across any host in any direction.

However, I can run iperf3 test in both directions and get speed around 200mbit/sec. If I run an iperf3 test while transferring files at sub 500kB/s speed, the iperf3 test is slower. Around 50mbit/sec ish.

Any ideas on how to figure out whats going on here? I'm lost.

[meyergru]

1. The reason why iperf probably shows a higher performance is that it is able to use multiple TCP streams, which you may have selected by using -Px (see https://forum.opnsense.org/index.php?topic=42985.0, point 10).

2. wget, curl rsync and scp use a signle TCP stream, where problems induced that can be induced by double NAT and the Proxmox virtualisation layer come into play. The fact that the rsync starts out fast and then drops speaks for buffer overruns.

I would try to further reduce the MTU size to 1360 on both sides first, regardless of if the pings work with your current settings.

Did you use virtio networking on the OpnSense VM with the settings depicted here, including RSS in OpnSense and on the VM NIC settings? In this specific case, multiqueue on the VM's NIC might be harmful, as it can cause out-of-order packets.

If you passthru the NIC, the VM might not get all interrupts in due time.

If you have to option, try to use a bare metal OpnSense in site B to isolate virtualisation issues.

Wireguard uses UDP and does not benefit from TCP buffer algorithms, so you might also try to use traffic shaping to ensure that buffers are not overrun.