Poor upload trough Firewall

[kdwhgw]

Hello community,

We have a strange bandwidth problem.

We are using this https://www.deciso.com/product-catalog/dec2640/ appliance on a symmetrical 200Mbit/s internet access.

We have noticed poor upload speed for TCP single stream connections through the firewall.
OPNsense version is OPNsense 23.7.4-amd64

I have run some iperf tests. The setup is simple.
iperf server <--> router <--> opnsense <--> clients

From firewall to internet I get full speed with ~190Mbit/s.
Clients to the firewall also looks good at ~900Mbit/s.

But clients to internet only gets between 20 and 50Mbit/s in upload. No active shaper or IDS on the firewall. WAN and LAN are each separate interfaces on the firewall, no VLAN. Hardware features for the interfaces are disabled.
I have tried different clients in our LAN, always with the same result.
Download looks normal.

In a test with several streams, the full bandwidth comes through in total. In addition, UDP connections do not seem to be affected.

iperf direct from the firewall to internet

Code: [Select]

root@opns-hgw-inet:~ # iperf3 -c x.x.x.x -P 1
Connecting to host x.x.x.x, port 5201
[  5] local y.y.y.y port 2078 connected to x.x.x.x port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.85 MBytes  49.1 Mbits/sec    0    907 KBytes
[  5]   1.00-2.00   sec  22.2 MBytes   186 Mbits/sec    0   3.00 MBytes
[  5]   2.00-3.00   sec  22.2 MBytes   186 Mbits/sec    0   3.00 MBytes
[  5]   3.00-4.00   sec  22.2 MBytes   186 Mbits/sec    0   3.00 MBytes
[  5]   4.00-5.00   sec  22.3 MBytes   187 Mbits/sec    0   3.00 MBytes
[  5]   5.00-6.00   sec  22.0 MBytes   184 Mbits/sec    0   3.00 MBytes
[  5]   6.00-7.00   sec  22.4 MBytes   188 Mbits/sec    0   3.00 MBytes
[  5]   7.00-8.00   sec  22.2 MBytes   186 Mbits/sec    0   3.00 MBytes
[  5]   8.00-9.00   sec  22.2 MBytes   187 Mbits/sec    0   3.00 MBytes
[  5]   9.00-10.00  sec  22.3 MBytes   187 Mbits/sec    0   3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   206 MBytes   173 Mbits/sec    0             sender
[  5]   0.00-10.09  sec   206 MBytes   171 Mbits/sec                  receiver

iperf Done.
iperf single stream tcp from client to firewall

Code: [Select]

root@xxxxxxxx:~# iperf3 -c 192.168.11.254 -p 13236 -P 1
Connecting to host 192.168.11.254, port 13236
[  5] local 192.168.11.221 port 39366 connected to 192.168.11.254 port 13236
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   101 MBytes   850 Mbits/sec   28    216 KBytes
[  5]   1.00-2.00   sec   102 MBytes   855 Mbits/sec   26    137 KBytes
[  5]   2.00-3.00   sec   101 MBytes   849 Mbits/sec   28    143 KBytes
[  5]   3.00-4.00   sec   102 MBytes   853 Mbits/sec   14    228 KBytes
[  5]   4.00-5.00   sec   102 MBytes   852 Mbits/sec   13   77.8 KBytes
[  5]   5.00-6.00   sec   101 MBytes   851 Mbits/sec   38    103 KBytes
[  5]   6.00-7.00   sec   101 MBytes   849 Mbits/sec   19    195 KBytes
[  5]   7.00-8.00   sec   101 MBytes   849 Mbits/sec   18    188 KBytes
[  5]   8.00-9.00   sec   101 MBytes   850 Mbits/sec    9    201 KBytes
[  5]   9.00-10.00  sec   101 MBytes   849 Mbits/sec   16    260 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1014 MBytes   851 Mbits/sec  209             sender
[  5]   0.00-10.00  sec  1013 MBytes   850 Mbits/sec                  receiver

iperf Done.
iperf single stream tcp from client to internet

Code: [Select]

root@xxxxxxxx:~# iperf3 -c x.x.x.x -P 1
Connecting to host x.x.x.x, port 5201
[  5] local 192.168.11.221 port 44070 connected to x.x.x.x port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  12.4 MBytes   104 Mbits/sec    3    417 KBytes
[  5]   1.00-2.00   sec  10.0 MBytes  83.9 Mbits/sec    1    320 KBytes
[  5]   2.00-3.00   sec  6.25 MBytes  52.4 Mbits/sec    4    178 KBytes
[  5]   3.00-4.00   sec  2.50 MBytes  21.0 Mbits/sec    1    142 KBytes
[  5]   4.00-5.00   sec  3.75 MBytes  31.5 Mbits/sec    1    111 KBytes
[  5]   5.00-6.00   sec  2.50 MBytes  21.0 Mbits/sec    0    127 KBytes
[  5]   6.00-7.00   sec  3.75 MBytes  31.5 Mbits/sec    0    145 KBytes
[  5]   7.00-8.00   sec  3.75 MBytes  31.5 Mbits/sec    0    163 KBytes
[  5]   8.00-9.00   sec  3.75 MBytes  31.5 Mbits/sec    0    180 KBytes
[  5]   9.00-10.00  sec  5.00 MBytes  41.9 Mbits/sec    0    198 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  53.7 MBytes  45.0 Mbits/sec   10             sender
[  5]   0.00-10.04  sec  50.1 MBytes  41.9 Mbits/sec                  receiver

iperf Done.
iperf with 10 connections from client to internet

Code: [Select]

root@xxxxxxxx:~# iperf3 -c x.x.x.x -P 10
Connecting to host x.x.x.x, port 5201
[  5] local 192.168.11.221 port 51946 connected to x.x.x.x port 5201
[  7] local 192.168.11.221 port 51960 connected to x.x.x.x port 5201
[  9] local 192.168.11.221 port 51964 connected to x.x.x.x port 5201
[ 11] local 192.168.11.221 port 51980 connected to x.x.x.x port 5201
[ 13] local 192.168.11.221 port 51988 connected to x.x.x.x port 5201
[ 15] local 192.168.11.221 port 52000 connected to x.x.x.x port 5201
[ 17] local 192.168.11.221 port 52012 connected to x.x.x.x port 5201
[ 19] local 192.168.11.221 port 52028 connected to x.x.x.x port 5201
[ 21] local 192.168.11.221 port 52030 connected to x.x.x.x port 5201
[ 23] local 192.168.11.221 port 52036 connected to x.x.x.x port 5201
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  20.1 MBytes  16.9 Mbits/sec    9             sender
[  5]   0.00-10.10  sec  18.2 MBytes  15.1 Mbits/sec                  receiver
[  7]   0.00-10.00  sec  35.5 MBytes  29.8 Mbits/sec    5             sender
[  7]   0.00-10.10  sec  32.3 MBytes  26.8 Mbits/sec                  receiver
[  9]   0.00-10.00  sec  42.0 MBytes  35.2 Mbits/sec    2             sender
[  9]   0.00-10.10  sec  39.4 MBytes  32.7 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec  24.9 MBytes  20.8 Mbits/sec    6             sender
[ 11]   0.00-10.10  sec  22.6 MBytes  18.8 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec  19.6 MBytes  16.4 Mbits/sec    4             sender
[ 13]   0.00-10.10  sec  17.9 MBytes  14.8 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec  25.2 MBytes  21.2 Mbits/sec    9             sender
[ 15]   0.00-10.10  sec  22.8 MBytes  18.9 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec  22.9 MBytes  19.2 Mbits/sec    8             sender
[ 17]   0.00-10.10  sec  20.8 MBytes  17.3 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec  16.9 MBytes  14.2 Mbits/sec    6             sender
[ 19]   0.00-10.10  sec  15.0 MBytes  12.5 Mbits/sec                  receiver
[ 21]   0.00-10.00  sec  17.7 MBytes  14.9 Mbits/sec    2             sender
[ 21]   0.00-10.10  sec  16.8 MBytes  14.0 Mbits/sec                  receiver
[ 23]   0.00-10.00  sec  16.4 MBytes  13.7 Mbits/sec    4             sender
[ 23]   0.00-10.10  sec  14.8 MBytes  12.3 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec   241 MBytes   202 Mbits/sec   55             sender
[SUM]   0.00-10.10  sec   221 MBytes   183 Mbits/sec                  receiver

iperf Done.
iperf with udp single stream from client to internet

Code: [Select]

root@xxxxxxxx:~# iperf3 -c x.x.x.x -P 1 -u -b 180M
Connecting to host x.x.x.x, port 5201
[  5] local 192.168.11.221 port 52403 connected to x.x.x.x port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  21.4 MBytes   180 Mbits/sec  16059
[  5]   1.00-2.00   sec  21.5 MBytes   180 Mbits/sec  16072
[  5]   2.00-3.00   sec  21.5 MBytes   180 Mbits/sec  16072
[  5]   3.00-4.00   sec  21.5 MBytes   180 Mbits/sec  16071
[  5]   4.00-5.00   sec  21.5 MBytes   180 Mbits/sec  16071
[  5]   5.00-6.00   sec  21.5 MBytes   180 Mbits/sec  16072
[  5]   6.00-7.00   sec  21.5 MBytes   180 Mbits/sec  16071
[  5]   7.00-8.00   sec  21.5 MBytes   180 Mbits/sec  16072
[  5]   8.00-9.00   sec  21.5 MBytes   180 Mbits/sec  16070
[  5]   9.00-10.00  sec  21.5 MBytes   180 Mbits/sec  16072
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   215 MBytes   180 Mbits/sec  0.000 ms  0/160702 (0%)  sender
[  5]   0.00-10.08  sec   213 MBytes   177 Mbits/sec  0.029 ms  1133/160702 (0.71%)  receiver

iperf Done.

[dellarte]

Hi,

Did you find a solution ?

I'm facing the same problem with upload bandwidth but with 2.5Gb down / 700Mb up
I'm trying to host a service but the upload bandwidth ruins it.

[rchris494]

try these settings.
https://binaryimpulse.com/2022/11/opnsense-performance-tuning-for-multi-gigabit-internet/

i was able to get 10gb on my internal up and down. my internet isn't that great. 1200mb down and only 40mb up. but running those tuning for it. i did see a speed increase from a default install of opnsense. i was able to handle all the traffic i could throw at it without any noticeable loss in speeds. i have only been on opnsense for a couple of months coming from clearos that is completely dead but would handle full speeds without any problems. due to lack of updates for the past 2+ years i had to find a new firewall. after testing several router software out. opnsense was the winner over the others. due to having max thoughput, easy to follow and current setup guides. along with having ipv6 external support.

granted my system is beyond overkill
hpz840 with dual E5-2690 v4 cpus  160gb of ddr4 mem
1 Silicom PE2G6I35-CX 6-Port 1GBase-T Gigabit for my 10/100/1000 network connections
4 HPE 562SFP+ 10Gb dual port
1 nvidia m2000 gpu just because its slim and i could add in another network card
1 Intel I226 dual port 2.5gb