Greetings.
I am having this problem since installing OPNsense on 2013 and I haven't fixed it yet (partly also because the sites have to be online and there are not many opportunities to shut them down). There is Wireguard site to site tunnel configuration executed based on official documentation step by step. Overall it works, but there is speed problem on one site, but only on one direction (site1 upload). It looks like a standard MSS/MTU or clamping issue for Wireguard, but that has been checked numerous times and no difference has been seen. Also the tunables have been checked and modified with no effect. Also all HW offloading is turned off on both OPNsense VMs in its settings.
The main configuration is as follows:
SITE1 PROXMOX host -> OPNsense guest (and some more VMs using LAN vmbr(0)) vmbr1 -> WAN bridge / Wireguard roadwarrior setup with a pair of clients, that can also access SITE2 trough SITE1
SITE2 PROXMOX host -> OPNsense guest (and some more VMs using LAN vmbr(0)) vmbr1 -> WAN bridge
Running OPNsense 25.7.3_7-amd64 on both hosts, always updated in the same time. Both guests are connected with site to site link. Link itself works and latency is low. I'm gonna add some speed tests. Internet link speed is 250/250 on both sides. Running with multiple connections on iperf doesnt help. Copying files using SMB results in same speed.
root@site2:~# iperf3 -c 10.44.1.2
Connecting to host 10.44.1.2, port 5201
[ 5] local 10.33.0.2 port 40808 connected to 10.44.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 18.7 MBytes 157 Mbits/sec 1002 71.3 KBytes
[ 5] 1.00-2.00 sec 8.69 MBytes 72.9 Mbits/sec 47 61.0 KBytes
[ 5] 2.00-3.00 sec 3.48 MBytes 29.1 Mbits/sec 81 20.8 KBytes
[ 5] 3.00-4.00 sec 10.4 MBytes 87.5 Mbits/sec 70 48.0 KBytes
[ 5] 4.00-5.00 sec 7.82 MBytes 65.6 Mbits/sec 69 64.8 KBytes
[ 5] 5.00-6.00 sec 9.56 MBytes 80.2 Mbits/sec 61 51.9 KBytes
[ 5] 6.00-7.00 sec 8.69 MBytes 72.9 Mbits/sec 52 31.1 KBytes
[ 5] 7.00-8.00 sec 7.82 MBytes 65.6 Mbits/sec 37 70.0 KBytes
[ 5] 8.00-9.00 sec 8.69 MBytes 72.9 Mbits/sec 50 71.3 KBytes
[ 5] 9.00-10.00 sec 9.56 MBytes 80.2 Mbits/sec 69 68.7 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 93.4 MBytes 78.3 Mbits/sec 1538 sender
[ 5] 0.00-10.01 sec 91.0 MBytes 76.3 Mbits/sec receiver
___
root@site1:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.33.0.2, port 40800
[ 5] local 10.44.1.2 port 5201 connected to 10.33.0.2 port 40808
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 16.6 MBytes 140 Mbits/sec
[ 5] 1.00-2.00 sec 8.79 MBytes 73.7 Mbits/sec
[ 5] 2.00-3.00 sec 3.64 MBytes 30.6 Mbits/sec
[ 5] 3.00-4.00 sec 9.99 MBytes 83.8 Mbits/sec
[ 5] 4.00-5.00 sec 7.64 MBytes 64.1 Mbits/sec
[ 5] 5.00-6.00 sec 9.60 MBytes 80.5 Mbits/sec
[ 5] 6.00-7.00 sec 8.59 MBytes 72.1 Mbits/sec
[ 5] 7.00-8.00 sec 8.57 MBytes 71.9 Mbits/sec
[ 5] 8.00-9.00 sec 8.37 MBytes 70.2 Mbits/sec
[ 5] 9.00-10.00 sec 9.06 MBytes 76.0 Mbits/sec
[ 5] 10.00-10.01 sec 53.2 KBytes 80.1 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 91.0 MBytes 76.3 Mbits/sec receiver
And then the other way:
root@site1:~# iperf3 -c 10.33.0.2
Connecting to host 10.33.0.2, port 5201
[ 5] local 10.44.1.2 port 42628 connected to 10.33.0.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 32.4 MBytes 272 Mbits/sec 100 908 KBytes
[ 5] 1.00-2.00 sec 23.8 MBytes 199 Mbits/sec 422 175 KBytes
[ 5] 2.00-3.00 sec 26.2 MBytes 220 Mbits/sec 54 122 KBytes
[ 5] 3.00-4.00 sec 28.8 MBytes 241 Mbits/sec 49 81.7 KBytes
[ 5] 4.00-5.00 sec 26.2 MBytes 220 Mbits/sec 0 208 KBytes
[ 5] 5.00-6.00 sec 26.2 MBytes 220 Mbits/sec 54 170 KBytes
[ 5] 6.00-7.00 sec 18.8 MBytes 157 Mbits/sec 69 66.1 KBytes
[ 5] 7.00-8.00 sec 25.0 MBytes 210 Mbits/sec 0 197 KBytes
[ 5] 8.00-9.00 sec 25.0 MBytes 210 Mbits/sec 58 170 KBytes
[ 5] 9.00-10.00 sec 25.0 MBytes 210 Mbits/sec 62 113 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 257 MBytes 216 Mbits/sec 868 sender
[ 5] 0.00-10.01 sec 255 MBytes 214 Mbits/sec receiver
root@site2:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.44.1.2, port 42614
[ 5] local 10.33.0.2 port 5201 connected to 10.44.1.2 port 42628
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 28.8 MBytes 242 Mbits/sec
[ 5] 1.00-2.00 sec 24.3 MBytes 204 Mbits/sec
[ 5] 2.00-3.00 sec 26.0 MBytes 218 Mbits/sec
[ 5] 3.00-4.00 sec 29.4 MBytes 246 Mbits/sec
[ 5] 4.00-5.00 sec 26.2 MBytes 220 Mbits/sec
[ 5] 5.00-6.00 sec 25.4 MBytes 213 Mbits/sec
[ 5] 6.00-7.00 sec 19.3 MBytes 162 Mbits/sec
[ 5] 7.00-8.00 sec 24.7 MBytes 207 Mbits/sec
[ 5] 8.00-9.00 sec 24.6 MBytes 206 Mbits/sec
[ 5] 9.00-10.00 sec 26.1 MBytes 219 Mbits/sec
[ 5] 10.00-10.01 sec 154 KBytes 218 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 255 MBytes 214 Mbits/sec receiver
On SITE1 there is also roadwarrior configuration, which also lets to access SITE2 local network. That all also works fine, just the speed problem. We can also see that the retry count is high when its throttling. When using roadwarrior on SITE1, we can see the exact same behavior (for example when trying speedtest.net test on a remote computer using SITE1 gateway). When utilizing internet (no WG) on both sites locally, I can get full 250/250 speed. That would seem that the problem is most likely only on SITE1 side and when using Wireguard.
The LAN segment on both sides works perfectly utilizing 10G link.
Then there is the fact, that WAN is configured using Proxmox virtual bridge (vmbr1) instead of using the real WAN interface as slave (vmbr0 localnet works fine with full passtrough). That is so on both servers. I have tried disabling every hardware offload function (tso,gso etc...) with ethtool -K on Proxmox hosts. Some entries (when checking with ethtool -k) show off [fixed] or off [forced on] even after trying to turn off. I have also read through this forum multiple times with no real solution. I have disabled 'Reply-To' in firewall settings and maybe some other tweaks that I forgot while writing this topic.
I am looking for an opportunity to enable iommu and pass the real wan interface trough Proxmox to the OPNsense VM's. Then attach it directly as WAN interface, but haven't tried it yet. I tried to fix is using the current setup, but it should be theoretically possible to work using current setup. Maybe someone can give some more ideas, or I should just try passing the PCI network adapter directly...
K