Hello community,
We have a strange bandwidth problem.
We are using this https://www.deciso.com/product-catalog/dec2640/ (https://www.deciso.com/product-catalog/dec2640/) appliance on a symmetrical 200Mbit/s internet access.
We have noticed poor upload speed for TCP single stream connections through the firewall.
OPNsense version is OPNsense 23.7.4-amd64
I have run some iperf tests. The setup is simple.
iperf server <--> router <--> opnsense <--> clients
From firewall to internet I get full speed with ~190Mbit/s.
Clients to the firewall also looks good at ~900Mbit/s.
But clients to internet only gets between 20 and 50Mbit/s in upload. No active shaper or IDS on the firewall. WAN and LAN are each separate interfaces on the firewall, no VLAN. Hardware features for the interfaces are disabled.
I have tried different clients in our LAN, always with the same result.
Download looks normal.
In a test with several streams, the full bandwidth comes through in total. In addition, UDP connections do not seem to be affected.
iperf direct from the firewall to internet
root@opns-hgw-inet:~ # iperf3 -c x.x.x.x -P 1
Connecting to host x.x.x.x, port 5201
[ 5] local y.y.y.y port 2078 connected to x.x.x.x port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 5.85 MBytes 49.1 Mbits/sec 0 907 KBytes
[ 5] 1.00-2.00 sec 22.2 MBytes 186 Mbits/sec 0 3.00 MBytes
[ 5] 2.00-3.00 sec 22.2 MBytes 186 Mbits/sec 0 3.00 MBytes
[ 5] 3.00-4.00 sec 22.2 MBytes 186 Mbits/sec 0 3.00 MBytes
[ 5] 4.00-5.00 sec 22.3 MBytes 187 Mbits/sec 0 3.00 MBytes
[ 5] 5.00-6.00 sec 22.0 MBytes 184 Mbits/sec 0 3.00 MBytes
[ 5] 6.00-7.00 sec 22.4 MBytes 188 Mbits/sec 0 3.00 MBytes
[ 5] 7.00-8.00 sec 22.2 MBytes 186 Mbits/sec 0 3.00 MBytes
[ 5] 8.00-9.00 sec 22.2 MBytes 187 Mbits/sec 0 3.00 MBytes
[ 5] 9.00-10.00 sec 22.3 MBytes 187 Mbits/sec 0 3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 206 MBytes 173 Mbits/sec 0 sender
[ 5] 0.00-10.09 sec 206 MBytes 171 Mbits/sec receiver
iperf Done.
iperf single stream tcp from client to firewall
root@xxxxxxxx:~# iperf3 -c 192.168.11.254 -p 13236 -P 1
Connecting to host 192.168.11.254, port 13236
[ 5] local 192.168.11.221 port 39366 connected to 192.168.11.254 port 13236
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 101 MBytes 850 Mbits/sec 28 216 KBytes
[ 5] 1.00-2.00 sec 102 MBytes 855 Mbits/sec 26 137 KBytes
[ 5] 2.00-3.00 sec 101 MBytes 849 Mbits/sec 28 143 KBytes
[ 5] 3.00-4.00 sec 102 MBytes 853 Mbits/sec 14 228 KBytes
[ 5] 4.00-5.00 sec 102 MBytes 852 Mbits/sec 13 77.8 KBytes
[ 5] 5.00-6.00 sec 101 MBytes 851 Mbits/sec 38 103 KBytes
[ 5] 6.00-7.00 sec 101 MBytes 849 Mbits/sec 19 195 KBytes
[ 5] 7.00-8.00 sec 101 MBytes 849 Mbits/sec 18 188 KBytes
[ 5] 8.00-9.00 sec 101 MBytes 850 Mbits/sec 9 201 KBytes
[ 5] 9.00-10.00 sec 101 MBytes 849 Mbits/sec 16 260 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1014 MBytes 851 Mbits/sec 209 sender
[ 5] 0.00-10.00 sec 1013 MBytes 850 Mbits/sec receiver
iperf Done.
iperf single stream tcp from client to internet
root@xxxxxxxx:~# iperf3 -c x.x.x.x -P 1
Connecting to host x.x.x.x, port 5201
[ 5] local 192.168.11.221 port 44070 connected to x.x.x.x port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 12.4 MBytes 104 Mbits/sec 3 417 KBytes
[ 5] 1.00-2.00 sec 10.0 MBytes 83.9 Mbits/sec 1 320 KBytes
[ 5] 2.00-3.00 sec 6.25 MBytes 52.4 Mbits/sec 4 178 KBytes
[ 5] 3.00-4.00 sec 2.50 MBytes 21.0 Mbits/sec 1 142 KBytes
[ 5] 4.00-5.00 sec 3.75 MBytes 31.5 Mbits/sec 1 111 KBytes
[ 5] 5.00-6.00 sec 2.50 MBytes 21.0 Mbits/sec 0 127 KBytes
[ 5] 6.00-7.00 sec 3.75 MBytes 31.5 Mbits/sec 0 145 KBytes
[ 5] 7.00-8.00 sec 3.75 MBytes 31.5 Mbits/sec 0 163 KBytes
[ 5] 8.00-9.00 sec 3.75 MBytes 31.5 Mbits/sec 0 180 KBytes
[ 5] 9.00-10.00 sec 5.00 MBytes 41.9 Mbits/sec 0 198 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 53.7 MBytes 45.0 Mbits/sec 10 sender
[ 5] 0.00-10.04 sec 50.1 MBytes 41.9 Mbits/sec receiver
iperf Done.
iperf with 10 connections from client to internet
root@xxxxxxxx:~# iperf3 -c x.x.x.x -P 10
Connecting to host x.x.x.x, port 5201
[ 5] local 192.168.11.221 port 51946 connected to x.x.x.x port 5201
[ 7] local 192.168.11.221 port 51960 connected to x.x.x.x port 5201
[ 9] local 192.168.11.221 port 51964 connected to x.x.x.x port 5201
[ 11] local 192.168.11.221 port 51980 connected to x.x.x.x port 5201
[ 13] local 192.168.11.221 port 51988 connected to x.x.x.x port 5201
[ 15] local 192.168.11.221 port 52000 connected to x.x.x.x port 5201
[ 17] local 192.168.11.221 port 52012 connected to x.x.x.x port 5201
[ 19] local 192.168.11.221 port 52028 connected to x.x.x.x port 5201
[ 21] local 192.168.11.221 port 52030 connected to x.x.x.x port 5201
[ 23] local 192.168.11.221 port 52036 connected to x.x.x.x port 5201
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 20.1 MBytes 16.9 Mbits/sec 9 sender
[ 5] 0.00-10.10 sec 18.2 MBytes 15.1 Mbits/sec receiver
[ 7] 0.00-10.00 sec 35.5 MBytes 29.8 Mbits/sec 5 sender
[ 7] 0.00-10.10 sec 32.3 MBytes 26.8 Mbits/sec receiver
[ 9] 0.00-10.00 sec 42.0 MBytes 35.2 Mbits/sec 2 sender
[ 9] 0.00-10.10 sec 39.4 MBytes 32.7 Mbits/sec receiver
[ 11] 0.00-10.00 sec 24.9 MBytes 20.8 Mbits/sec 6 sender
[ 11] 0.00-10.10 sec 22.6 MBytes 18.8 Mbits/sec receiver
[ 13] 0.00-10.00 sec 19.6 MBytes 16.4 Mbits/sec 4 sender
[ 13] 0.00-10.10 sec 17.9 MBytes 14.8 Mbits/sec receiver
[ 15] 0.00-10.00 sec 25.2 MBytes 21.2 Mbits/sec 9 sender
[ 15] 0.00-10.10 sec 22.8 MBytes 18.9 Mbits/sec receiver
[ 17] 0.00-10.00 sec 22.9 MBytes 19.2 Mbits/sec 8 sender
[ 17] 0.00-10.10 sec 20.8 MBytes 17.3 Mbits/sec receiver
[ 19] 0.00-10.00 sec 16.9 MBytes 14.2 Mbits/sec 6 sender
[ 19] 0.00-10.10 sec 15.0 MBytes 12.5 Mbits/sec receiver
[ 21] 0.00-10.00 sec 17.7 MBytes 14.9 Mbits/sec 2 sender
[ 21] 0.00-10.10 sec 16.8 MBytes 14.0 Mbits/sec receiver
[ 23] 0.00-10.00 sec 16.4 MBytes 13.7 Mbits/sec 4 sender
[ 23] 0.00-10.10 sec 14.8 MBytes 12.3 Mbits/sec receiver
[SUM] 0.00-10.00 sec 241 MBytes 202 Mbits/sec 55 sender
[SUM] 0.00-10.10 sec 221 MBytes 183 Mbits/sec receiver
iperf Done.
iperf with udp single stream from client to internet
root@xxxxxxxx:~# iperf3 -c x.x.x.x -P 1 -u -b 180M
Connecting to host x.x.x.x, port 5201
[ 5] local 192.168.11.221 port 52403 connected to x.x.x.x port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 21.4 MBytes 180 Mbits/sec 16059
[ 5] 1.00-2.00 sec 21.5 MBytes 180 Mbits/sec 16072
[ 5] 2.00-3.00 sec 21.5 MBytes 180 Mbits/sec 16072
[ 5] 3.00-4.00 sec 21.5 MBytes 180 Mbits/sec 16071
[ 5] 4.00-5.00 sec 21.5 MBytes 180 Mbits/sec 16071
[ 5] 5.00-6.00 sec 21.5 MBytes 180 Mbits/sec 16072
[ 5] 6.00-7.00 sec 21.5 MBytes 180 Mbits/sec 16071
[ 5] 7.00-8.00 sec 21.5 MBytes 180 Mbits/sec 16072
[ 5] 8.00-9.00 sec 21.5 MBytes 180 Mbits/sec 16070
[ 5] 9.00-10.00 sec 21.5 MBytes 180 Mbits/sec 16072
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 215 MBytes 180 Mbits/sec 0.000 ms 0/160702 (0%) sender
[ 5] 0.00-10.08 sec 213 MBytes 177 Mbits/sec 0.029 ms 1133/160702 (0.71%) receiver
iperf Done.
Hi,
Did you find a solution ?
I'm facing the same problem with upload bandwidth but with 2.5Gb down / 700Mb up
I'm trying to host a service but the upload bandwidth ruins it.
try these settings.
https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/
i was able to get 10gb on my internal up and down. my internet isn't that great. 1200mb down and only 40mb up. but running those tuning for it. i did see a speed increase from a default install of opnsense. i was able to handle all the traffic i could throw at it without any noticeable loss in speeds. i have only been on opnsense for a couple of months coming from clearos that is completely dead but would handle full speeds without any problems. due to lack of updates for the past 2+ years i had to find a new firewall. after testing several router software out. opnsense was the winner over the others. due to having max thoughput, easy to follow and current setup guides. along with having ipv6 external support.
granted my system is beyond overkill
hpz840 with dual E5-2690 v4 cpus 160gb of ddr4 mem
1 Silicom PE2G6I35-CX 6-Port 1GBase-T Gigabit for my 10/100/1000 network connections
4 HPE 562SFP+ 10Gb dual port
1 nvidia m2000 gpu just because its slim and i could add in another network card
1 Intel I226 dual port 2.5gb