Slow throughput performance ?

[Kamikaze]

I´m running a Sophos SG230 with a I3-4130T CPU on a Deutsche Glasfaser / German Fiber with a 1000/500 MBit bandwith.
An IPerf3 test from the Sophos to ping.online.net gives these results:

Code Select Expand

root@OPNsense:/home/remote_access # iperf3 -R -P 1 -c ping.online.net
Connecting to host ping.online.net, port 5201
Reverse mode, remote host ping.online.net is sending
[  5] local x.x.x.x port 11897 connected to 51.158.1.21 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  41.9 MBytes   348 Mbits/sec
[  5]   1.01-2.00   sec  60.0 MBytes   507 Mbits/sec
[  5]   2.00-3.00   sec  60.4 MBytes   506 Mbits/sec
[  5]   3.00-4.00   sec  60.0 MBytes   503 Mbits/sec
[  5]   4.00-5.01   sec  60.9 MBytes   506 Mbits/sec
[  5]   5.01-6.01   sec  60.1 MBytes   504 Mbits/sec
[  5]   6.01-7.00   sec  60.0 MBytes   507 Mbits/sec
[  5]   7.00-8.02   sec  61.1 MBytes   507 Mbits/sec
[  5]   8.02-9.00   sec  60.1 MBytes   511 Mbits/sec
[  5]   9.00-10.00  sec  60.8 MBytes   510 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec   620 MBytes   519 Mbits/sec    0            sender
[  5]   0.00-10.00  sec   585 MBytes   491 Mbits/sec                  receiver


Code Select Expand

root@OPNsense:/home/remote_access # iperf3 -R -P 10 -c ping.online.net
Connecting to host ping.online.net, port 5201
Reverse mode, remote host ping.online.net is sending
[  5] local x.x.x.x port 41516 connected to 51.158.1.21 port 5201
[  7] local x.x.x.x port 21762 connected to 51.158.1.21 port 5201
[  9] local x.x.x.x port 40228 connected to 51.158.1.21 port 5201
[ 11] local x.x.x.x port 58922 connected to 51.158.1.21 port 5201
[ 13] local x.x.x.x port 8851 connected to 51.158.1.21 port 5201
[ 15] local x.x.x.x port 38318 connected to 51.158.1.21 port 5201
[ 17] local x.x.x.x port 20949 connected to 51.158.1.21 port 5201
[ 19] local x.x.x.x port 28493 connected to 51.158.1.21 port 5201
[ 21] local x.x.x.x port 21965 connected to 51.158.1.21 port 5201
[ 23] local x.x.x.x port 51096 connected to 51.158.1.21 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.03   sec  12.6 MBytes   102 Mbits/sec
[  7]   0.00-1.04   sec  5.38 MBytes  43.6 Mbits/sec
[  9]   0.00-1.04   sec  8.12 MBytes  65.9 Mbits/sec
[ 11]   0.00-1.04   sec  5.00 MBytes  40.5 Mbits/sec
[ 13]   0.00-1.04   sec  7.50 MBytes  60.8 Mbits/sec
[ 15]   0.00-1.04   sec  11.1 MBytes  90.2 Mbits/sec
[ 17]   0.00-1.04   sec  5.25 MBytes  42.5 Mbits/sec
[ 19]   0.00-1.04   sec  7.38 MBytes  59.8 Mbits/sec
[ 21]   0.00-1.04   sec  9.50 MBytes  77.0 Mbits/sec
[ 23]   0.00-1.04   sec  5.50 MBytes  44.6 Mbits/sec
[SUM]   0.00-1.03   sec  77.4 MBytes   627 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   151 MBytes   126 Mbits/sec  11374            sender
[  5]   0.00-10.00  sec   136 MBytes   114 Mbits/sec                  receiver
[  7]   0.00-10.04  sec  73.8 MBytes  61.7 Mbits/sec  4144            sender
[  7]   0.00-10.00  sec  65.6 MBytes  55.0 Mbits/sec                  receiver
[  9]   0.00-10.04  sec   107 MBytes  89.2 Mbits/sec  6748            sender
[  9]   0.00-10.00  sec  97.5 MBytes  81.8 Mbits/sec                  receiver
[ 11]   0.00-10.04  sec  71.2 MBytes  59.5 Mbits/sec  3744            sender
[ 11]   0.00-10.00  sec  65.1 MBytes  54.6 Mbits/sec                  receiver
[ 13]   0.00-10.04  sec   114 MBytes  95.0 Mbits/sec  8341            sender
[ 13]   0.00-10.00  sec   103 MBytes  86.5 Mbits/sec                  receiver
[ 15]   0.00-10.04  sec   155 MBytes   130 Mbits/sec  10877            sender
[ 15]   0.00-10.00  sec   141 MBytes   118 Mbits/sec                  receiver
[ 17]   0.00-10.04  sec  76.3 MBytes  63.8 Mbits/sec  4158            sender
[ 17]   0.00-10.00  sec  67.1 MBytes  56.3 Mbits/sec                  receiver
[ 19]   0.00-10.04  sec   104 MBytes  87.2 Mbits/sec  7275            sender
[ 19]   0.00-10.00  sec  95.2 MBytes  79.9 Mbits/sec                  receiver
[ 21]   0.00-10.04  sec   143 MBytes   119 Mbits/sec  9469            sender
[ 21]   0.00-10.00  sec   130 MBytes   109 Mbits/sec                  receiver
[ 23]   0.00-10.04  sec  71.2 MBytes  59.5 Mbits/sec  4243            sender
[ 23]   0.00-10.00  sec  64.9 MBytes  54.4 Mbits/sec                  receiver
[SUM]   0.00-10.04  sec  1.04 GBytes   891 Mbits/sec  70373             sender
[SUM]   0.00-10.00  sec   965 MBytes   809 Mbits/sec

The Iperf3 from my client to the Sophos gives these here:

Sophos => Client => as expected around 850Mbits

Code Select Expand

iperf3.exe -c 192.168.1.1 -R -p 57426

Connecting to host 192.168.1.1, port 57426
Reverse mode, remote host 192.168.1.1 is sending
[  5] local 192.168.1.90 port 62588 connected to 192.168.1.1 port 57426
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   110 MBytes   911 Mbits/sec
[  5]   1.01-2.01   sec   106 MBytes   894 Mbits/sec
[  5]   2.01-3.01   sec  99.9 MBytes   833 Mbits/sec
[  5]   3.01-4.01   sec  98.9 MBytes   832 Mbits/sec
[  5]   4.01-5.00   sec   104 MBytes   875 Mbits/sec
[  5]   5.00-6.00   sec  90.2 MBytes   758 Mbits/sec
[  5]   6.00-7.01   sec   106 MBytes   884 Mbits/sec
[  5]   7.01-8.01   sec   105 MBytes   882 Mbits/sec
[  5]   8.01-9.01   sec   102 MBytes   852 Mbits/sec
[  5]   9.01-10.00  sec   106 MBytes   893 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.00 GBytes   861 Mbits/sec    0            sender
[  5]   0.00-10.00  sec  1.00 GBytes   862 Mbits/sec                  receiver

Client => Sohpos => The first oddity - its only around 200-250Mbits

Code Select Expand

iperf3.exe -c 192.168.1.1 -p 1734
Connecting to host 192.168.1.1, port 1734
[  5] local 192.168.1.90 port 62615 connected to 192.168.1.1 port 1734
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  35.6 MBytes   298 Mbits/sec
[  5]   1.00-2.00   sec  26.2 MBytes   220 Mbits/sec
[  5]   2.00-3.00   sec  25.0 MBytes   210 Mbits/sec
[  5]   3.00-4.01   sec  22.1 MBytes   183 Mbits/sec
[  5]   4.01-5.01   sec  23.0 MBytes   194 Mbits/sec
[  5]   5.01-6.01   sec  17.5 MBytes   147 Mbits/sec
[  5]   6.01-7.00   sec  22.6 MBytes   191 Mbits/sec
[  5]   7.00-8.02   sec  20.4 MBytes   169 Mbits/sec
[  5]   8.02-9.01   sec  17.8 MBytes   149 Mbits/sec
[  5]   9.01-10.01  sec  20.2 MBytes   171 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   230 MBytes   193 Mbits/sec                  sender
[  5]   0.00-10.01  sec   230 MBytes   193 Mbits/sec                  receiver

When i run a Iperf to the online.net server it looks like this:

Code Select Expand

iperf3.exe -c ping.online.net -R -P 10
Connecting to host ping.online.net, port 5201
Reverse mode, remote host ping.online.net is sending
[  5] local 192.168.1.90 port 52456 connected to 51.158.1.21 port 5201
[  7] local 192.168.1.90 port 52457 connected to 51.158.1.21 port 5201
[  9] local 192.168.1.90 port 52458 connected to 51.158.1.21 port 5201
[ 11] local 192.168.1.90 port 52459 connected to 51.158.1.21 port 5201
[ 13] local 192.168.1.90 port 52460 connected to 51.158.1.21 port 5201
[ 15] local 192.168.1.90 port 52461 connected to 51.158.1.21 port 5201
[ 17] local 192.168.1.90 port 52462 connected to 51.158.1.21 port 5201
[ 19] local 192.168.1.90 port 52463 connected to 51.158.1.21 port 5201
[ 21] local 192.168.1.90 port 52464 connected to 51.158.1.21 port 5201
[ 23] local 192.168.1.90 port 52465 connected to 51.158.1.21 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.02   sec  1.75 MBytes  14.5 Mbits/sec
[  7]   0.00-1.02   sec  1.75 MBytes  14.5 Mbits/sec
[  9]   0.00-1.02   sec  1.62 MBytes  13.4 Mbits/sec
[ 11]   0.00-1.02   sec  1.75 MBytes  14.5 Mbits/sec
[ 13]   0.00-1.02   sec  1.62 MBytes  13.4 Mbits/sec
[ 15]   0.00-1.02   sec  1.62 MBytes  13.4 Mbits/sec
[ 17]   0.00-1.02   sec  1.62 MBytes  13.4 Mbits/sec
[ 19]   0.00-1.02   sec  1.62 MBytes  13.4 Mbits/sec
[ 21]   0.00-1.02   sec  1.50 MBytes  12.4 Mbits/sec
[ 23]   0.00-1.02   sec  1.62 MBytes  13.4 Mbits/sec
[SUM]   0.00-1.02   sec  16.5 MBytes   136 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  19.1 MBytes  16.0 Mbits/sec    0            sender
[  5]   0.00-10.00  sec  17.8 MBytes  14.9 Mbits/sec                  receiver
[  7]   0.00-10.04  sec  19.1 MBytes  15.9 Mbits/sec    0            sender
[  7]   0.00-10.00  sec  17.8 MBytes  14.9 Mbits/sec                  receiver
[  9]   0.00-10.04  sec  19.0 MBytes  15.9 Mbits/sec    0            sender
[  9]   0.00-10.00  sec  17.6 MBytes  14.8 Mbits/sec                  receiver
[ 11]   0.00-10.04  sec  19.1 MBytes  15.9 Mbits/sec    0            sender
[ 11]   0.00-10.00  sec  17.9 MBytes  15.0 Mbits/sec                  receiver
[ 13]   0.00-10.04  sec  18.1 MBytes  15.2 Mbits/sec    0            sender
[ 13]   0.00-10.00  sec  16.9 MBytes  14.2 Mbits/sec                  receiver
[ 15]   0.00-10.04  sec  19.0 MBytes  15.9 Mbits/sec    0            sender
[ 15]   0.00-10.00  sec  17.6 MBytes  14.8 Mbits/sec                  receiver
[ 17]   0.00-10.04  sec  18.3 MBytes  15.3 Mbits/sec    0            sender
[ 17]   0.00-10.00  sec  16.6 MBytes  13.9 Mbits/sec                  receiver
[ 19]   0.00-10.04  sec  18.1 MBytes  15.1 Mbits/sec    0            sender
[ 19]   0.00-10.00  sec  16.6 MBytes  13.9 Mbits/sec                  receiver
[ 21]   0.00-10.04  sec  18.0 MBytes  15.0 Mbits/sec    0            sender
[ 21]   0.00-10.00  sec  16.4 MBytes  13.7 Mbits/sec                  receiver
[ 23]   0.00-10.04  sec  18.0 MBytes  15.1 Mbits/sec    0            sender
[ 23]   0.00-10.00  sec  16.6 MBytes  13.9 Mbits/sec                  receiver
[SUM]   0.00-10.04  sec   186 MBytes   155 Mbits/sec    0             sender
[SUM]   0.00-10.00  sec   172 MBytes   144 Mbits/sec                  receiver
[code]

Thats abound 20% of the same test as onlinet.net => Sophos

The other way the same - also only 20% ...

[code]
iperf3.exe -c ping.online.net -P 10
Connecting to host ping.online.net, port 5201
[  5] local 192.168.1.90 port 53910 connected to 51.158.1.21 port 5201
[  7] local 192.168.1.90 port 53911 connected to 51.158.1.21 port 5201
[  9] local 192.168.1.90 port 53912 connected to 51.158.1.21 port 5201
[ 11] local 192.168.1.90 port 53913 connected to 51.158.1.21 port 5201
[ 13] local 192.168.1.90 port 53914 connected to 51.158.1.21 port 5201
[ 15] local 192.168.1.90 port 53915 connected to 51.158.1.21 port 5201
[ 17] local 192.168.1.90 port 53916 connected to 51.158.1.21 port 5201
[ 19] local 192.168.1.90 port 53917 connected to 51.158.1.21 port 5201
[ 21] local 192.168.1.90 port 53918 connected to 51.158.1.21 port 5201
[ 23] local 192.168.1.90 port 53919 connected to 51.158.1.21 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  1.75 MBytes  14.5 Mbits/sec
[  7]   0.00-1.01   sec  1.75 MBytes  14.5 Mbits/sec
[  9]   0.00-1.01   sec  1.75 MBytes  14.5 Mbits/sec
[ 11]   0.00-1.01   sec  1.75 MBytes  14.5 Mbits/sec
[ 13]   0.00-1.01   sec  1.75 MBytes  14.5 Mbits/sec
[ 15]   0.00-1.01   sec  1.75 MBytes  14.5 Mbits/sec
[ 17]   0.00-1.01   sec  1.62 MBytes  13.4 Mbits/sec
[ 19]   0.00-1.01   sec  1.75 MBytes  14.5 Mbits/sec
[ 21]   0.00-1.01   sec  1.62 MBytes  13.4 Mbits/sec
[ 23]   0.00-1.01   sec  1.62 MBytes  13.4 Mbits/sec
[SUM]   0.00-1.01   sec  17.1 MBytes   142 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.01-2.01   sec  1.88 MBytes  15.8 Mbits/sec
[  7]   1.01-2.01   sec  1.88 MBytes  15.8 Mbits/sec
[  9]   1.01-2.01   sec  1.62 MBytes  13.7 Mbits/sec
[ 11]   1.01-2.01   sec  1.88 MBytes  15.8 Mbits/sec
[ 13]   1.01-2.01   sec  1.62 MBytes  13.7 Mbits/sec
[ 15]   1.01-2.01   sec  1.75 MBytes  14.7 Mbits/sec
[ 17]   1.01-2.01   sec  1.75 MBytes  14.7 Mbits/sec
[ 19]   1.01-2.01   sec  1.75 MBytes  14.7 Mbits/sec
[ 21]   1.01-2.01   sec  1.75 MBytes  14.7 Mbits/sec
[ 23]   1.01-2.01   sec  1.75 MBytes  14.7 Mbits/sec
[SUM]   1.01-2.01   sec  17.6 MBytes   148 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  18.1 MBytes  15.2 Mbits/sec                  sender
[  5]   0.00-10.04  sec  18.0 MBytes  15.1 Mbits/sec                  receiver
[  7]   0.00-10.01  sec  18.1 MBytes  15.2 Mbits/sec                  sender
[  7]   0.00-10.04  sec  18.0 MBytes  15.1 Mbits/sec                  receiver
[  9]   0.00-10.01  sec  17.1 MBytes  14.4 Mbits/sec                  sender
[  9]   0.00-10.04  sec  17.0 MBytes  14.2 Mbits/sec                  receiver
[ 11]   0.00-10.01  sec  18.1 MBytes  15.2 Mbits/sec                  sender
[ 11]   0.00-10.04  sec  18.0 MBytes  15.0 Mbits/sec                  receiver
[ 13]   0.00-10.01  sec  17.0 MBytes  14.3 Mbits/sec                  sender
[ 13]   0.00-10.04  sec  16.9 MBytes  14.1 Mbits/sec                  receiver
[ 15]   0.00-10.01  sec  17.2 MBytes  14.5 Mbits/sec                  sender
[ 15]   0.00-10.04  sec  17.1 MBytes  14.3 Mbits/sec                  receiver
[ 17]   0.00-10.01  sec  16.8 MBytes  14.0 Mbits/sec                  sender
[ 17]   0.00-10.04  sec  16.6 MBytes  13.8 Mbits/sec                  receiver
[ 19]   0.00-10.01  sec  17.5 MBytes  14.7 Mbits/sec                  sender
[ 19]   0.00-10.04  sec  17.4 MBytes  14.5 Mbits/sec                  receiver
[ 21]   0.00-10.01  sec  17.1 MBytes  14.4 Mbits/sec                  sender
[ 21]   0.00-10.04  sec  17.0 MBytes  14.2 Mbits/sec                  receiver
[ 23]   0.00-10.01  sec  17.4 MBytes  14.6 Mbits/sec                  sender
[ 23]   0.00-10.04  sec  17.2 MBytes  14.4 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec   174 MBytes   146 Mbits/sec                  sender
[SUM]   0.00-10.04  sec   173 MBytes   145 Mbits/sec                  receiver

Does anybody have any idea ?
I havent setup any firewall rules except for the most basic ones ...

[Seimus]

First of all, this is not routing performance but throughput performance.

Second what is your CPU utilization when you see performance degradation?
Are you running any IPS or ZenArmor?

Regards,
S.

[pfry]

Hey! I have one of those: HDPlex H10.

I don't have much specific to say about it. OPNsense is reasonably well optimized by default, but you could try further optimizations (lots of threads on that here). The CPU is a fixed 2.9GHz dual-core Haswell, so there's a limit to what you can squeeze out of it.

[Kamikaze]

First of all, this is not routing performance but throughput performance.

Second what is your CPU utilization when you see performance degradation?
Are you running any IPS or ZenArmor?

Regards,
S.

The Maximum I´ve seen ( even while running Iperf ) was around 0,3 in the Dashboard.
I dont use any IPS and ZenArmor - at least not that I know of ( relativly new to OPNsense, I´ve used pfSense and OpenWRT before ).

[Seimus]

If this OPNsense out of the box than you have no IPS or ZenArmor preinstalled. Basically you have just lean Packet Firewall.

That 0,3 I believe you mean load? And whats the utilization?
Also what NICs has the Sophos, intel or realtek?

Regards,
S.

[Kamikaze]

If this OPNsense out of the box than you have no IPS or ZenArmor preinstalled. Basically you have just lean Packet Firewall.

That 0,3 I believe you mean load? And whats the utilization?
Also what NICs has the Sophos, intel or realtek?

Regards,
S.

The NICs are Intel, they show up as IGB0-5 ( I use 0 for LAN, 5 for WAN ).
Regarding the utilization - i only have the dashboard widget as reference - any other way to get the info ?

[pfry]

[...]
Regarding the utilization - i only have the dashboard widget as reference - any other way to get the info ?

One would be "top", from a console/terminal. But the widget (as of v25 or so) has options to display System/User/Interrupt/Total, so it's not a bad general view. It just won't report on specific processes.

[Kamikaze]

[...]
Regarding the utilization - i only have the dashboard widget as reference - any other way to get the info ?

One would be "top", from a console/terminal. But the widget (as of v25 or so) has options to display System/User/Interrupt/Total, so it's not a bad general view. It just won't report on specific processes.

during a Public IPerf => Sophos test:
Total hat a spike at 35
Interrupt to 9
User 22
System 17

[Kamikaze]

Update:
I installed my old OpenWRT router that was running fine on our 1000/50 cable.
Absolutely same results and that's with completly different hardware even with client directly connected to the router and even different clients.

The moment the traffic goes to the LAN and not the external IP of the router and inet connection becomes really slow.

Is that throttling from the ISP ?

[Kamikaze]

SOLVED !
We ran multiple test over the weekend again, even it turned out that the client we testet on were the problem.
When we used a linux live stick the performance was as expected and from there on the issue were found with the tcp autotuning.
after running these commands the bandwith exploded:

netsh int tcp set global autotuninglevel=normal
netsh int tcp set global rss=enabled