OpenVPN DCO site-to-site speed issue

Started by fixinit75, July 31, 2025, 11:20:50 PM

Previous topic - Next topic
Print
Go Down Pages1
User actions
July 31, 2025, 11:20:50 PM
I have an internal network that consists of different locations with their networks interconnected with OpenVPN tunnels through one specific location used as a traffic exchange. I had used pfSense on all the locations that are owned or set up by me, but for some reasons (including that Netgate only offers the DCO support for the pfSense Plus edition and without it I get like 100 mbps speeds between the locations) I had to switch to OPNSense. The last one was that on a traffic exchange location. I have successfully switched it a day ago and have some troubles with speeds since then.

So, let's focus on one of the zones (we'll call it 'zone B') and the traffic exchange one (we'll call it 'zone A').
Zone A's OPNSense is a proxmox VM with 4 cores r7 5800x and 4 gigs of ram. It is connected to WAN almost directly: it is connected to virtio network device, which is connected to the WAN NIC, no other VMs are connected to this interface, proxmox also isn't. WAN NIC is connected to the ISP. Firewall also has numerous interfaces, including LAN for the VMs.
Zone B's OPNSense is a physical device (IE-AP300), which has an Intel Atom E3940 4-core CPU and 10 gigs of ram. It is also directly connected to the ISP on WAN side and to the gigabit switch on the LAN side.
The networks, as mentioned above, are interconnected with an OpenVPN tunnel. In the zone A I have an Ubuntu VM connected to the LAN of the firewall. In the zone B I have a Windows PC connected to the firewall through a switch.

When I connect the Windows PC directly to the zone A's firewall with OpenVPN and use DCO, I get the speeds of ~50-62 MiB/s (tested with sftp between the ubuntu vm), which is the maximum speed of the internet I get from the ISP of the zone B (500 mbps). If I disable DCO, I get the speeds of around 20-30 MiB/s (~250-300 mbps).
But when I use the VPN tunnel between the zones, I get the speeds of 1.5-2 MiB/s (shows as 40 mbps in windows taskmgr) with DCO enabled. Sadly, I couldn't measure the speeds without DCO as the routing between the networks just stops working for some reason when I disable DCO.
I don't see high CPU load on any of the firewalls, and this problem happens on every location that has OPNSense with OpenVPN DCO tunnels. One of them even measured 22 mbps download and 0.45 mbps upload speeds.

I am rather new to the OPNSense system, so I may have missed something.

Here is the iperf measurement between the zones B and A:

root@OPNsense:~ # iperf3 --client 100.64.10.1 -p 4064 --no-delay --parallel 8
Connecting to host 100.64.10.1, port 4064
[  5] local 10.0.13.2 port 42842 connected to 100.64.10.1 port 4064
[  7] local 10.0.13.2 port 3671 connected to 100.64.10.1 port 4064
[  9] local 10.0.13.2 port 40419 connected to 100.64.10.1 port 4064
[ 11] local 10.0.13.2 port 47190 connected to 100.64.10.1 port 4064
[ 13] local 10.0.13.2 port 5983 connected to 100.64.10.1 port 4064
[ 15] local 10.0.13.2 port 9582 connected to 100.64.10.1 port 4064
[ 17] local 10.0.13.2 port 47254 connected to 100.64.10.1 port 4064
[ 19] local 10.0.13.2 port 7662 connected to 100.64.10.1 port 4064
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.02   sec   128 KBytes  1.02 Mbits/sec   35   0.00 Bytes
[  7]   0.00-1.02   sec   384 KBytes  3.07 Mbits/sec   41   1.41 KBytes
[  9]   0.00-1.02   sec   640 KBytes  5.12 Mbits/sec   62   1.41 KBytes
[ 11]   0.00-1.02   sec   256 KBytes  2.05 Mbits/sec   45   2.83 KBytes
[ 13]   0.00-1.02   sec   512 KBytes  4.10 Mbits/sec   64   1.41 KBytes
[ 15]   0.00-1.02   sec   128 KBytes  1.02 Mbits/sec   33   0.00 Bytes
[ 17]   0.00-1.02   sec   128 KBytes  1.02 Mbits/sec   28   0.00 Bytes
[ 19]   0.00-1.02   sec   384 KBytes  3.07 Mbits/sec   30   2.83 KBytes
[SUM]   0.00-1.02   sec  2.50 MBytes  20.5 Mbits/sec  338
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.02-2.00   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[  7]   1.02-2.00   sec   384 KBytes  3.22 Mbits/sec   23   8.48 KBytes
[  9]   1.02-2.00   sec   512 KBytes  4.29 Mbits/sec   32   7.95 KBytes
[ 11]   1.02-2.00   sec   384 KBytes  3.22 Mbits/sec   14   5.88 KBytes
[ 13]   1.02-2.00   sec   640 KBytes  5.36 Mbits/sec   51   15.1 KBytes
[ 15]   1.02-2.00   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[ 17]   1.02-2.00   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[ 19]   1.02-2.00   sec   768 KBytes  6.43 Mbits/sec   48   15.6 KBytes
[SUM]   1.02-2.00   sec  2.62 MBytes  22.5 Mbits/sec  168
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.01   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[  7]   2.00-3.01   sec   384 KBytes  3.13 Mbits/sec   18   8.27 KBytes
[  9]   2.00-3.01   sec   384 KBytes  3.13 Mbits/sec   35   9.90 KBytes
[ 11]   2.00-3.01   sec   384 KBytes  3.13 Mbits/sec   32   17.6 KBytes
[ 13]   2.00-3.01   sec   640 KBytes  5.22 Mbits/sec   35   18.2 KBytes
[ 15]   2.00-3.01   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[ 17]   2.00-3.01   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[ 19]   2.00-3.01   sec   512 KBytes  4.17 Mbits/sec   43   17.0 KBytes
[SUM]   2.00-3.01   sec  2.25 MBytes  18.8 Mbits/sec  163
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.01-4.04   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[  7]   3.01-4.04   sec   256 KBytes  2.02 Mbits/sec   15   8.48 KBytes
[  9]   3.01-4.04   sec   384 KBytes  3.03 Mbits/sec   25   17.0 KBytes
[ 11]   3.01-4.04   sec   512 KBytes  4.04 Mbits/sec   41   19.0 KBytes
[ 13]   3.01-4.04   sec   512 KBytes  4.04 Mbits/sec   35   20.7 KBytes
[ 15]   3.01-4.04   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[ 17]   3.01-4.04   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[ 19]   3.01-4.04   sec   512 KBytes  4.04 Mbits/sec   35   20.6 KBytes
[SUM]   3.01-4.04   sec  2.12 MBytes  17.2 Mbits/sec  151
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.04-5.00   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[  7]   4.04-5.00   sec   256 KBytes  2.19 Mbits/sec   19   5.66 KBytes
[  9]   4.04-5.00   sec   256 KBytes  2.19 Mbits/sec   48   10.1 KBytes
[ 11]   4.04-5.00   sec   512 KBytes  4.38 Mbits/sec   44   11.3 KBytes
[ 13]   4.04-5.00   sec   384 KBytes  3.28 Mbits/sec   47   11.3 KBytes
[ 15]   4.04-5.00   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[ 17]   4.04-5.00   sec  0.00 Bytes  0.00 bits/sec    0   0.00 Bytes
[ 19]   4.04-5.00   sec   512 KBytes  4.38 Mbits/sec   49   11.3 KBytes
[SUM]   4.04-5.00   sec  1.88 MBytes  16.4 Mbits/sec  207
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.02   sec   128 KBytes  1.03 Mbits/sec    7   11.2 KBytes
[  7]   5.00-6.02   sec   128 KBytes  1.03 Mbits/sec   15   6.59 KBytes
[  9]   5.00-6.02   sec   384 KBytes  3.10 Mbits/sec   32   13.7 KBytes
[ 11]   5.00-6.02   sec   384 KBytes  3.10 Mbits/sec   35   15.2 KBytes
[ 13]   5.00-6.02   sec   512 KBytes  4.13 Mbits/sec   34   14.2 KBytes
[ 15]   5.00-6.02   sec   384 KBytes  3.10 Mbits/sec   11   11.3 KBytes
[ 17]   5.00-6.02   sec   384 KBytes  3.10 Mbits/sec   10   9.90 KBytes
[ 19]   5.00-6.02   sec   512 KBytes  4.13 Mbits/sec   35   14.1 KBytes
[SUM]   5.00-6.02   sec  2.75 MBytes  22.7 Mbits/sec  179
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.02-7.00   sec   256 KBytes  2.13 Mbits/sec   24   13.3 KBytes
[  7]   6.02-7.00   sec   256 KBytes  2.13 Mbits/sec   13   6.54 KBytes
[  9]   6.02-7.00   sec   384 KBytes  3.19 Mbits/sec   28   1.41 KBytes
[ 11]   6.02-7.00   sec   256 KBytes  2.13 Mbits/sec   30   1.41 KBytes
[ 13]   6.02-7.00   sec   256 KBytes  2.13 Mbits/sec   29   15.5 KBytes
[ 15]   6.02-7.00   sec   256 KBytes  2.13 Mbits/sec   19   7.09 KBytes
[ 17]   6.02-7.00   sec   128 KBytes  1.06 Mbits/sec   16   5.66 KBytes
[ 19]   6.02-7.00   sec   256 KBytes  2.13 Mbits/sec   27   15.2 KBytes
[SUM]   6.02-7.00   sec  2.00 MBytes  17.0 Mbits/sec  186
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.04   sec   256 KBytes  2.01 Mbits/sec   47   12.9 KBytes
[  7]   7.00-8.04   sec   128 KBytes  1.01 Mbits/sec   17   6.25 KBytes
[  9]   7.00-8.04   sec   384 KBytes  3.02 Mbits/sec   38   12.9 KBytes
[ 11]   7.00-8.04   sec   384 KBytes  3.02 Mbits/sec   43   13.8 KBytes
[ 13]   7.00-8.04   sec   384 KBytes  3.02 Mbits/sec   37   7.85 KBytes
[ 15]   7.00-8.04   sec   256 KBytes  2.01 Mbits/sec   56   14.2 KBytes
[ 17]   7.00-8.04   sec   256 KBytes  2.01 Mbits/sec   15   5.13 KBytes
[ 19]   7.00-8.04   sec   384 KBytes  3.02 Mbits/sec   39   13.8 KBytes
[SUM]   7.00-8.04   sec  2.38 MBytes  19.1 Mbits/sec  292
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.04-9.02   sec   256 KBytes  2.14 Mbits/sec   30   14.9 KBytes
[  7]   8.04-9.02   sec   256 KBytes  2.14 Mbits/sec   14   1.41 KBytes
[  9]   8.04-9.02   sec   256 KBytes  2.14 Mbits/sec   28   1.41 KBytes
[ 11]   8.04-9.02   sec   384 KBytes  3.22 Mbits/sec   33   1.41 KBytes
[ 13]   8.04-9.02   sec   256 KBytes  2.14 Mbits/sec   28   1.41 KBytes
[ 15]   8.04-9.02   sec   256 KBytes  2.14 Mbits/sec   29   2.83 KBytes
[ 17]   8.04-9.02   sec   128 KBytes  1.07 Mbits/sec   10   1.41 KBytes
[ 19]   8.04-9.02   sec   256 KBytes  2.14 Mbits/sec   27   16.6 KBytes
[SUM]   8.04-9.02   sec  2.00 MBytes  17.1 Mbits/sec  199
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.02-10.03  sec   384 KBytes  3.12 Mbits/sec   37   18.4 KBytes
[  7]   9.02-10.03  sec   128 KBytes  1.04 Mbits/sec   15   7.80 KBytes
[  9]   9.02-10.03  sec   384 KBytes  3.12 Mbits/sec   36   15.4 KBytes
[ 11]   9.02-10.03  sec   384 KBytes  3.12 Mbits/sec   45   18.0 KBytes
[ 13]   9.02-10.03  sec   384 KBytes  3.12 Mbits/sec   42   17.0 KBytes
[ 15]   9.02-10.03  sec   384 KBytes  3.12 Mbits/sec   29   14.3 KBytes
[ 17]   9.02-10.03  sec   128 KBytes  1.04 Mbits/sec   13   6.51 KBytes
[ 19]   9.02-10.03  sec   384 KBytes  3.12 Mbits/sec   46   18.4 KBytes
[SUM]   9.02-10.03  sec  2.50 MBytes  20.8 Mbits/sec  263
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec  1.38 MBytes  1.15 Mbits/sec  180            sender
[  5]   0.00-10.04  sec  1.38 MBytes  1.15 Mbits/sec                  receiver
[  7]   0.00-10.03  sec  2.50 MBytes  2.09 Mbits/sec  190            sender
[  7]   0.00-10.04  sec  2.38 MBytes  1.99 Mbits/sec                  receiver
[  9]   0.00-10.03  sec  3.88 MBytes  3.24 Mbits/sec  364            sender
[  9]   0.00-10.04  sec  3.75 MBytes  3.13 Mbits/sec                  receiver
[ 11]   0.00-10.03  sec  3.75 MBytes  3.14 Mbits/sec  362            sender
[ 11]   0.00-10.04  sec  3.62 MBytes  3.03 Mbits/sec                  receiver
[ 13]   0.00-10.03  sec  4.38 MBytes  3.66 Mbits/sec  402            sender
[ 13]   0.00-10.04  sec  4.25 MBytes  3.55 Mbits/sec                  receiver
[ 15]   0.00-10.03  sec  1.62 MBytes  1.36 Mbits/sec  177            sender
[ 15]   0.00-10.04  sec  1.50 MBytes  1.25 Mbits/sec                  receiver
[ 17]   0.00-10.03  sec  1.12 MBytes   941 Kbits/sec   92            sender
[ 17]   0.00-10.04  sec  1.00 MBytes   836 Kbits/sec                  receiver
[ 19]   0.00-10.03  sec  4.38 MBytes  3.66 Mbits/sec  379            sender
[ 19]   0.00-10.04  sec  4.38 MBytes  3.66 Mbits/sec                  receiver
[SUM]   0.00-10.03  sec  23.0 MBytes  19.2 Mbits/sec  2146             sender
[SUM]   0.00-10.04  sec  22.2 MBytes  18.6 Mbits/sec                  receiver

iperf Done.

P.S. I know I should never use carrier-grade NAT IPs in my networks, but my ISPs use 192.168.X.X IPs instead of these, so that's ok for me and has no effect on the speed test results
Print
Go Up Pages1
User actions