I have an internal network that consists of different locations with their networks interconnected with OpenVPN tunnels through one specific location used as a traffic exchange. I had used pfSense on all the locations that are owned or set up by me, but for some reasons (including that Netgate only offers the DCO support for the pfSense Plus edition and without it I get like 100 mbps speeds between the locations) I had to switch to OPNSense. The last one was that on a traffic exchange location. I have successfully switched it a day ago and have some troubles with speeds since then.
So, let's focus on one of the zones (we'll call it 'zone B') and the traffic exchange one (we'll call it 'zone A').
Zone A's OPNSense is a proxmox VM with 4 cores r7 5800x and 4 gigs of ram. It is connected to WAN almost directly: it is connected to virtio network device, which is connected to the WAN NIC, no other VMs are connected to this interface, proxmox also isn't. WAN NIC is connected to the ISP. Firewall also has numerous interfaces, including LAN for the VMs.
Zone B's OPNSense is a physical device (IE-AP300), which has an Intel Atom E3940 4-core CPU and 10 gigs of ram. It is also directly connected to the ISP on WAN side and to the gigabit switch on the LAN side.
The networks, as mentioned above, are interconnected with an OpenVPN tunnel. In the zone A I have an Ubuntu VM connected to the LAN of the firewall. In the zone B I have a Windows PC connected to the firewall through a switch.
When I connect the Windows PC directly to the zone A's firewall with OpenVPN and use DCO, I get the speeds of ~50-62 MiB/s (tested with sftp between the ubuntu vm), which is the maximum speed of the internet I get from the ISP of the zone B (500 mbps). If I disable DCO, I get the speeds of around 20-30 MiB/s (~250-300 mbps).
But when I use the VPN tunnel between the zones, I get the speeds of 1.5-2 MiB/s (shows as 40 mbps in windows taskmgr) with DCO enabled. Sadly, I couldn't measure the speeds without DCO as the routing between the networks just stops working for some reason when I disable DCO.
I don't see high CPU load on any of the firewalls, and this problem happens on every location that has OPNSense with OpenVPN DCO tunnels. One of them even measured 22 mbps download and 0.45 mbps upload speeds.
I am rather new to the OPNSense system, so I may have missed something.
Here is the iperf measurement between the zones B and A:
root@OPNsense:~ # iperf3 --client 100.64.10.1 -p 4064 --no-delay --parallel 8
Connecting to host 100.64.10.1, port 4064
[ 5] local 10.0.13.2 port 42842 connected to 100.64.10.1 port 4064
[ 7] local 10.0.13.2 port 3671 connected to 100.64.10.1 port 4064
[ 9] local 10.0.13.2 port 40419 connected to 100.64.10.1 port 4064
[ 11] local 10.0.13.2 port 47190 connected to 100.64.10.1 port 4064
[ 13] local 10.0.13.2 port 5983 connected to 100.64.10.1 port 4064
[ 15] local 10.0.13.2 port 9582 connected to 100.64.10.1 port 4064
[ 17] local 10.0.13.2 port 47254 connected to 100.64.10.1 port 4064
[ 19] local 10.0.13.2 port 7662 connected to 100.64.10.1 port 4064
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.02 sec 128 KBytes 1.02 Mbits/sec 35 0.00 Bytes
[ 7] 0.00-1.02 sec 384 KBytes 3.07 Mbits/sec 41 1.41 KBytes
[ 9] 0.00-1.02 sec 640 KBytes 5.12 Mbits/sec 62 1.41 KBytes
[ 11] 0.00-1.02 sec 256 KBytes 2.05 Mbits/sec 45 2.83 KBytes
[ 13] 0.00-1.02 sec 512 KBytes 4.10 Mbits/sec 64 1.41 KBytes
[ 15] 0.00-1.02 sec 128 KBytes 1.02 Mbits/sec 33 0.00 Bytes
[ 17] 0.00-1.02 sec 128 KBytes 1.02 Mbits/sec 28 0.00 Bytes
[ 19] 0.00-1.02 sec 384 KBytes 3.07 Mbits/sec 30 2.83 KBytes
[SUM] 0.00-1.02 sec 2.50 MBytes 20.5 Mbits/sec 338
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.02-2.00 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 7] 1.02-2.00 sec 384 KBytes 3.22 Mbits/sec 23 8.48 KBytes
[ 9] 1.02-2.00 sec 512 KBytes 4.29 Mbits/sec 32 7.95 KBytes
[ 11] 1.02-2.00 sec 384 KBytes 3.22 Mbits/sec 14 5.88 KBytes
[ 13] 1.02-2.00 sec 640 KBytes 5.36 Mbits/sec 51 15.1 KBytes
[ 15] 1.02-2.00 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 17] 1.02-2.00 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 19] 1.02-2.00 sec 768 KBytes 6.43 Mbits/sec 48 15.6 KBytes
[SUM] 1.02-2.00 sec 2.62 MBytes 22.5 Mbits/sec 168
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.01 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 7] 2.00-3.01 sec 384 KBytes 3.13 Mbits/sec 18 8.27 KBytes
[ 9] 2.00-3.01 sec 384 KBytes 3.13 Mbits/sec 35 9.90 KBytes
[ 11] 2.00-3.01 sec 384 KBytes 3.13 Mbits/sec 32 17.6 KBytes
[ 13] 2.00-3.01 sec 640 KBytes 5.22 Mbits/sec 35 18.2 KBytes
[ 15] 2.00-3.01 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 17] 2.00-3.01 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 19] 2.00-3.01 sec 512 KBytes 4.17 Mbits/sec 43 17.0 KBytes
[SUM] 2.00-3.01 sec 2.25 MBytes 18.8 Mbits/sec 163
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.01-4.04 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 7] 3.01-4.04 sec 256 KBytes 2.02 Mbits/sec 15 8.48 KBytes
[ 9] 3.01-4.04 sec 384 KBytes 3.03 Mbits/sec 25 17.0 KBytes
[ 11] 3.01-4.04 sec 512 KBytes 4.04 Mbits/sec 41 19.0 KBytes
[ 13] 3.01-4.04 sec 512 KBytes 4.04 Mbits/sec 35 20.7 KBytes
[ 15] 3.01-4.04 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 17] 3.01-4.04 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 19] 3.01-4.04 sec 512 KBytes 4.04 Mbits/sec 35 20.6 KBytes
[SUM] 3.01-4.04 sec 2.12 MBytes 17.2 Mbits/sec 151
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.04-5.00 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 7] 4.04-5.00 sec 256 KBytes 2.19 Mbits/sec 19 5.66 KBytes
[ 9] 4.04-5.00 sec 256 KBytes 2.19 Mbits/sec 48 10.1 KBytes
[ 11] 4.04-5.00 sec 512 KBytes 4.38 Mbits/sec 44 11.3 KBytes
[ 13] 4.04-5.00 sec 384 KBytes 3.28 Mbits/sec 47 11.3 KBytes
[ 15] 4.04-5.00 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 17] 4.04-5.00 sec 0.00 Bytes 0.00 bits/sec 0 0.00 Bytes
[ 19] 4.04-5.00 sec 512 KBytes 4.38 Mbits/sec 49 11.3 KBytes
[SUM] 4.04-5.00 sec 1.88 MBytes 16.4 Mbits/sec 207
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.02 sec 128 KBytes 1.03 Mbits/sec 7 11.2 KBytes
[ 7] 5.00-6.02 sec 128 KBytes 1.03 Mbits/sec 15 6.59 KBytes
[ 9] 5.00-6.02 sec 384 KBytes 3.10 Mbits/sec 32 13.7 KBytes
[ 11] 5.00-6.02 sec 384 KBytes 3.10 Mbits/sec 35 15.2 KBytes
[ 13] 5.00-6.02 sec 512 KBytes 4.13 Mbits/sec 34 14.2 KBytes
[ 15] 5.00-6.02 sec 384 KBytes 3.10 Mbits/sec 11 11.3 KBytes
[ 17] 5.00-6.02 sec 384 KBytes 3.10 Mbits/sec 10 9.90 KBytes
[ 19] 5.00-6.02 sec 512 KBytes 4.13 Mbits/sec 35 14.1 KBytes
[SUM] 5.00-6.02 sec 2.75 MBytes 22.7 Mbits/sec 179
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.02-7.00 sec 256 KBytes 2.13 Mbits/sec 24 13.3 KBytes
[ 7] 6.02-7.00 sec 256 KBytes 2.13 Mbits/sec 13 6.54 KBytes
[ 9] 6.02-7.00 sec 384 KBytes 3.19 Mbits/sec 28 1.41 KBytes
[ 11] 6.02-7.00 sec 256 KBytes 2.13 Mbits/sec 30 1.41 KBytes
[ 13] 6.02-7.00 sec 256 KBytes 2.13 Mbits/sec 29 15.5 KBytes
[ 15] 6.02-7.00 sec 256 KBytes 2.13 Mbits/sec 19 7.09 KBytes
[ 17] 6.02-7.00 sec 128 KBytes 1.06 Mbits/sec 16 5.66 KBytes
[ 19] 6.02-7.00 sec 256 KBytes 2.13 Mbits/sec 27 15.2 KBytes
[SUM] 6.02-7.00 sec 2.00 MBytes 17.0 Mbits/sec 186
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.04 sec 256 KBytes 2.01 Mbits/sec 47 12.9 KBytes
[ 7] 7.00-8.04 sec 128 KBytes 1.01 Mbits/sec 17 6.25 KBytes
[ 9] 7.00-8.04 sec 384 KBytes 3.02 Mbits/sec 38 12.9 KBytes
[ 11] 7.00-8.04 sec 384 KBytes 3.02 Mbits/sec 43 13.8 KBytes
[ 13] 7.00-8.04 sec 384 KBytes 3.02 Mbits/sec 37 7.85 KBytes
[ 15] 7.00-8.04 sec 256 KBytes 2.01 Mbits/sec 56 14.2 KBytes
[ 17] 7.00-8.04 sec 256 KBytes 2.01 Mbits/sec 15 5.13 KBytes
[ 19] 7.00-8.04 sec 384 KBytes 3.02 Mbits/sec 39 13.8 KBytes
[SUM] 7.00-8.04 sec 2.38 MBytes 19.1 Mbits/sec 292
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.04-9.02 sec 256 KBytes 2.14 Mbits/sec 30 14.9 KBytes
[ 7] 8.04-9.02 sec 256 KBytes 2.14 Mbits/sec 14 1.41 KBytes
[ 9] 8.04-9.02 sec 256 KBytes 2.14 Mbits/sec 28 1.41 KBytes
[ 11] 8.04-9.02 sec 384 KBytes 3.22 Mbits/sec 33 1.41 KBytes
[ 13] 8.04-9.02 sec 256 KBytes 2.14 Mbits/sec 28 1.41 KBytes
[ 15] 8.04-9.02 sec 256 KBytes 2.14 Mbits/sec 29 2.83 KBytes
[ 17] 8.04-9.02 sec 128 KBytes 1.07 Mbits/sec 10 1.41 KBytes
[ 19] 8.04-9.02 sec 256 KBytes 2.14 Mbits/sec 27 16.6 KBytes
[SUM] 8.04-9.02 sec 2.00 MBytes 17.1 Mbits/sec 199
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.02-10.03 sec 384 KBytes 3.12 Mbits/sec 37 18.4 KBytes
[ 7] 9.02-10.03 sec 128 KBytes 1.04 Mbits/sec 15 7.80 KBytes
[ 9] 9.02-10.03 sec 384 KBytes 3.12 Mbits/sec 36 15.4 KBytes
[ 11] 9.02-10.03 sec 384 KBytes 3.12 Mbits/sec 45 18.0 KBytes
[ 13] 9.02-10.03 sec 384 KBytes 3.12 Mbits/sec 42 17.0 KBytes
[ 15] 9.02-10.03 sec 384 KBytes 3.12 Mbits/sec 29 14.3 KBytes
[ 17] 9.02-10.03 sec 128 KBytes 1.04 Mbits/sec 13 6.51 KBytes
[ 19] 9.02-10.03 sec 384 KBytes 3.12 Mbits/sec 46 18.4 KBytes
[SUM] 9.02-10.03 sec 2.50 MBytes 20.8 Mbits/sec 263
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.03 sec 1.38 MBytes 1.15 Mbits/sec 180 sender
[ 5] 0.00-10.04 sec 1.38 MBytes 1.15 Mbits/sec receiver
[ 7] 0.00-10.03 sec 2.50 MBytes 2.09 Mbits/sec 190 sender
[ 7] 0.00-10.04 sec 2.38 MBytes 1.99 Mbits/sec receiver
[ 9] 0.00-10.03 sec 3.88 MBytes 3.24 Mbits/sec 364 sender
[ 9] 0.00-10.04 sec 3.75 MBytes 3.13 Mbits/sec receiver
[ 11] 0.00-10.03 sec 3.75 MBytes 3.14 Mbits/sec 362 sender
[ 11] 0.00-10.04 sec 3.62 MBytes 3.03 Mbits/sec receiver
[ 13] 0.00-10.03 sec 4.38 MBytes 3.66 Mbits/sec 402 sender
[ 13] 0.00-10.04 sec 4.25 MBytes 3.55 Mbits/sec receiver
[ 15] 0.00-10.03 sec 1.62 MBytes 1.36 Mbits/sec 177 sender
[ 15] 0.00-10.04 sec 1.50 MBytes 1.25 Mbits/sec receiver
[ 17] 0.00-10.03 sec 1.12 MBytes 941 Kbits/sec 92 sender
[ 17] 0.00-10.04 sec 1.00 MBytes 836 Kbits/sec receiver
[ 19] 0.00-10.03 sec 4.38 MBytes 3.66 Mbits/sec 379 sender
[ 19] 0.00-10.04 sec 4.38 MBytes 3.66 Mbits/sec receiver
[SUM] 0.00-10.03 sec 23.0 MBytes 19.2 Mbits/sec 2146 sender
[SUM] 0.00-10.04 sec 22.2 MBytes 18.6 Mbits/sec receiver
iperf Done.
P.S. I know I should never use carrier-grade NAT IPs in my networks, but my ISPs use 192.168.X.X IPs instead of these, so that's ok for me and has no effect on the speed test results