I´m running a Sophos SG230 with a I3-4130T CPU on a Deutsche Glasfaser / German Fiber with a 1000/500 MBit bandwith.
An IPerf3 test from the Sophos to ping.online.net gives these results:
root@OPNsense:/home/remote_access # iperf3 -R -P 1 -c ping.online.net
Connecting to host ping.online.net, port 5201
Reverse mode, remote host ping.online.net is sending
[ 5] local x.x.x.x port 11897 connected to 51.158.1.21 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 41.9 MBytes 348 Mbits/sec
[ 5] 1.01-2.00 sec 60.0 MBytes 507 Mbits/sec
[ 5] 2.00-3.00 sec 60.4 MBytes 506 Mbits/sec
[ 5] 3.00-4.00 sec 60.0 MBytes 503 Mbits/sec
[ 5] 4.00-5.01 sec 60.9 MBytes 506 Mbits/sec
[ 5] 5.01-6.01 sec 60.1 MBytes 504 Mbits/sec
[ 5] 6.01-7.00 sec 60.0 MBytes 507 Mbits/sec
[ 5] 7.00-8.02 sec 61.1 MBytes 507 Mbits/sec
[ 5] 8.02-9.00 sec 60.1 MBytes 511 Mbits/sec
[ 5] 9.00-10.00 sec 60.8 MBytes 510 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.03 sec 620 MBytes 519 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 585 MBytes 491 Mbits/sec receiver
root@OPNsense:/home/remote_access # iperf3 -R -P 10 -c ping.online.net
Connecting to host ping.online.net, port 5201
Reverse mode, remote host ping.online.net is sending
[ 5] local x.x.x.x port 41516 connected to 51.158.1.21 port 5201
[ 7] local x.x.x.x port 21762 connected to 51.158.1.21 port 5201
[ 9] local x.x.x.x port 40228 connected to 51.158.1.21 port 5201
[ 11] local x.x.x.x port 58922 connected to 51.158.1.21 port 5201
[ 13] local x.x.x.x port 8851 connected to 51.158.1.21 port 5201
[ 15] local x.x.x.x port 38318 connected to 51.158.1.21 port 5201
[ 17] local x.x.x.x port 20949 connected to 51.158.1.21 port 5201
[ 19] local x.x.x.x port 28493 connected to 51.158.1.21 port 5201
[ 21] local x.x.x.x port 21965 connected to 51.158.1.21 port 5201
[ 23] local x.x.x.x port 51096 connected to 51.158.1.21 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.03 sec 12.6 MBytes 102 Mbits/sec
[ 7] 0.00-1.04 sec 5.38 MBytes 43.6 Mbits/sec
[ 9] 0.00-1.04 sec 8.12 MBytes 65.9 Mbits/sec
[ 11] 0.00-1.04 sec 5.00 MBytes 40.5 Mbits/sec
[ 13] 0.00-1.04 sec 7.50 MBytes 60.8 Mbits/sec
[ 15] 0.00-1.04 sec 11.1 MBytes 90.2 Mbits/sec
[ 17] 0.00-1.04 sec 5.25 MBytes 42.5 Mbits/sec
[ 19] 0.00-1.04 sec 7.38 MBytes 59.8 Mbits/sec
[ 21] 0.00-1.04 sec 9.50 MBytes 77.0 Mbits/sec
[ 23] 0.00-1.04 sec 5.50 MBytes 44.6 Mbits/sec
[SUM] 0.00-1.03 sec 77.4 MBytes 627 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 151 MBytes 126 Mbits/sec 11374 sender
[ 5] 0.00-10.00 sec 136 MBytes 114 Mbits/sec receiver
[ 7] 0.00-10.04 sec 73.8 MBytes 61.7 Mbits/sec 4144 sender
[ 7] 0.00-10.00 sec 65.6 MBytes 55.0 Mbits/sec receiver
[ 9] 0.00-10.04 sec 107 MBytes 89.2 Mbits/sec 6748 sender
[ 9] 0.00-10.00 sec 97.5 MBytes 81.8 Mbits/sec receiver
[ 11] 0.00-10.04 sec 71.2 MBytes 59.5 Mbits/sec 3744 sender
[ 11] 0.00-10.00 sec 65.1 MBytes 54.6 Mbits/sec receiver
[ 13] 0.00-10.04 sec 114 MBytes 95.0 Mbits/sec 8341 sender
[ 13] 0.00-10.00 sec 103 MBytes 86.5 Mbits/sec receiver
[ 15] 0.00-10.04 sec 155 MBytes 130 Mbits/sec 10877 sender
[ 15] 0.00-10.00 sec 141 MBytes 118 Mbits/sec receiver
[ 17] 0.00-10.04 sec 76.3 MBytes 63.8 Mbits/sec 4158 sender
[ 17] 0.00-10.00 sec 67.1 MBytes 56.3 Mbits/sec receiver
[ 19] 0.00-10.04 sec 104 MBytes 87.2 Mbits/sec 7275 sender
[ 19] 0.00-10.00 sec 95.2 MBytes 79.9 Mbits/sec receiver
[ 21] 0.00-10.04 sec 143 MBytes 119 Mbits/sec 9469 sender
[ 21] 0.00-10.00 sec 130 MBytes 109 Mbits/sec receiver
[ 23] 0.00-10.04 sec 71.2 MBytes 59.5 Mbits/sec 4243 sender
[ 23] 0.00-10.00 sec 64.9 MBytes 54.4 Mbits/sec receiver
[SUM] 0.00-10.04 sec 1.04 GBytes 891 Mbits/sec 70373 sender
[SUM] 0.00-10.00 sec 965 MBytes 809 Mbits/sec
The Iperf3 from my client to the Sophos gives these here:
Sophos => Client => as expected around 850Mbits
iperf3.exe -c 192.168.1.1 -R -p 57426
Connecting to host 192.168.1.1, port 57426
Reverse mode, remote host 192.168.1.1 is sending
[ 5] local 192.168.1.90 port 62588 connected to 192.168.1.1 port 57426
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 110 MBytes 911 Mbits/sec
[ 5] 1.01-2.01 sec 106 MBytes 894 Mbits/sec
[ 5] 2.01-3.01 sec 99.9 MBytes 833 Mbits/sec
[ 5] 3.01-4.01 sec 98.9 MBytes 832 Mbits/sec
[ 5] 4.01-5.00 sec 104 MBytes 875 Mbits/sec
[ 5] 5.00-6.00 sec 90.2 MBytes 758 Mbits/sec
[ 5] 6.00-7.01 sec 106 MBytes 884 Mbits/sec
[ 5] 7.01-8.01 sec 105 MBytes 882 Mbits/sec
[ 5] 8.01-9.01 sec 102 MBytes 852 Mbits/sec
[ 5] 9.01-10.00 sec 106 MBytes 893 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 1.00 GBytes 861 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.00 GBytes 862 Mbits/sec receiver
Client => Sohpos => The first oddity - its only around 200-250Mbits
iperf3.exe -c 192.168.1.1 -p 1734
Connecting to host 192.168.1.1, port 1734
[ 5] local 192.168.1.90 port 62615 connected to 192.168.1.1 port 1734
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 35.6 MBytes 298 Mbits/sec
[ 5] 1.00-2.00 sec 26.2 MBytes 220 Mbits/sec
[ 5] 2.00-3.00 sec 25.0 MBytes 210 Mbits/sec
[ 5] 3.00-4.01 sec 22.1 MBytes 183 Mbits/sec
[ 5] 4.01-5.01 sec 23.0 MBytes 194 Mbits/sec
[ 5] 5.01-6.01 sec 17.5 MBytes 147 Mbits/sec
[ 5] 6.01-7.00 sec 22.6 MBytes 191 Mbits/sec
[ 5] 7.00-8.02 sec 20.4 MBytes 169 Mbits/sec
[ 5] 8.02-9.01 sec 17.8 MBytes 149 Mbits/sec
[ 5] 9.01-10.01 sec 20.2 MBytes 171 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 230 MBytes 193 Mbits/sec sender
[ 5] 0.00-10.01 sec 230 MBytes 193 Mbits/sec receiver
When i run a Iperf to the online.net server it looks like this:
iperf3.exe -c ping.online.net -R -P 10
Connecting to host ping.online.net, port 5201
Reverse mode, remote host ping.online.net is sending
[ 5] local 192.168.1.90 port 52456 connected to 51.158.1.21 port 5201
[ 7] local 192.168.1.90 port 52457 connected to 51.158.1.21 port 5201
[ 9] local 192.168.1.90 port 52458 connected to 51.158.1.21 port 5201
[ 11] local 192.168.1.90 port 52459 connected to 51.158.1.21 port 5201
[ 13] local 192.168.1.90 port 52460 connected to 51.158.1.21 port 5201
[ 15] local 192.168.1.90 port 52461 connected to 51.158.1.21 port 5201
[ 17] local 192.168.1.90 port 52462 connected to 51.158.1.21 port 5201
[ 19] local 192.168.1.90 port 52463 connected to 51.158.1.21 port 5201
[ 21] local 192.168.1.90 port 52464 connected to 51.158.1.21 port 5201
[ 23] local 192.168.1.90 port 52465 connected to 51.158.1.21 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.02 sec 1.75 MBytes 14.5 Mbits/sec
[ 7] 0.00-1.02 sec 1.75 MBytes 14.5 Mbits/sec
[ 9] 0.00-1.02 sec 1.62 MBytes 13.4 Mbits/sec
[ 11] 0.00-1.02 sec 1.75 MBytes 14.5 Mbits/sec
[ 13] 0.00-1.02 sec 1.62 MBytes 13.4 Mbits/sec
[ 15] 0.00-1.02 sec 1.62 MBytes 13.4 Mbits/sec
[ 17] 0.00-1.02 sec 1.62 MBytes 13.4 Mbits/sec
[ 19] 0.00-1.02 sec 1.62 MBytes 13.4 Mbits/sec
[ 21] 0.00-1.02 sec 1.50 MBytes 12.4 Mbits/sec
[ 23] 0.00-1.02 sec 1.62 MBytes 13.4 Mbits/sec
[SUM] 0.00-1.02 sec 16.5 MBytes 136 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 19.1 MBytes 16.0 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 17.8 MBytes 14.9 Mbits/sec receiver
[ 7] 0.00-10.04 sec 19.1 MBytes 15.9 Mbits/sec 0 sender
[ 7] 0.00-10.00 sec 17.8 MBytes 14.9 Mbits/sec receiver
[ 9] 0.00-10.04 sec 19.0 MBytes 15.9 Mbits/sec 0 sender
[ 9] 0.00-10.00 sec 17.6 MBytes 14.8 Mbits/sec receiver
[ 11] 0.00-10.04 sec 19.1 MBytes 15.9 Mbits/sec 0 sender
[ 11] 0.00-10.00 sec 17.9 MBytes 15.0 Mbits/sec receiver
[ 13] 0.00-10.04 sec 18.1 MBytes 15.2 Mbits/sec 0 sender
[ 13] 0.00-10.00 sec 16.9 MBytes 14.2 Mbits/sec receiver
[ 15] 0.00-10.04 sec 19.0 MBytes 15.9 Mbits/sec 0 sender
[ 15] 0.00-10.00 sec 17.6 MBytes 14.8 Mbits/sec receiver
[ 17] 0.00-10.04 sec 18.3 MBytes 15.3 Mbits/sec 0 sender
[ 17] 0.00-10.00 sec 16.6 MBytes 13.9 Mbits/sec receiver
[ 19] 0.00-10.04 sec 18.1 MBytes 15.1 Mbits/sec 0 sender
[ 19] 0.00-10.00 sec 16.6 MBytes 13.9 Mbits/sec receiver
[ 21] 0.00-10.04 sec 18.0 MBytes 15.0 Mbits/sec 0 sender
[ 21] 0.00-10.00 sec 16.4 MBytes 13.7 Mbits/sec receiver
[ 23] 0.00-10.04 sec 18.0 MBytes 15.1 Mbits/sec 0 sender
[ 23] 0.00-10.00 sec 16.6 MBytes 13.9 Mbits/sec receiver
[SUM] 0.00-10.04 sec 186 MBytes 155 Mbits/sec 0 sender
[SUM] 0.00-10.00 sec 172 MBytes 144 Mbits/sec receiver
[code]
Thats abound 20% of the same test as onlinet.net => Sophos
The other way the same - also only 20% ...
[code]
iperf3.exe -c ping.online.net -P 10
Connecting to host ping.online.net, port 5201
[ 5] local 192.168.1.90 port 53910 connected to 51.158.1.21 port 5201
[ 7] local 192.168.1.90 port 53911 connected to 51.158.1.21 port 5201
[ 9] local 192.168.1.90 port 53912 connected to 51.158.1.21 port 5201
[ 11] local 192.168.1.90 port 53913 connected to 51.158.1.21 port 5201
[ 13] local 192.168.1.90 port 53914 connected to 51.158.1.21 port 5201
[ 15] local 192.168.1.90 port 53915 connected to 51.158.1.21 port 5201
[ 17] local 192.168.1.90 port 53916 connected to 51.158.1.21 port 5201
[ 19] local 192.168.1.90 port 53917 connected to 51.158.1.21 port 5201
[ 21] local 192.168.1.90 port 53918 connected to 51.158.1.21 port 5201
[ 23] local 192.168.1.90 port 53919 connected to 51.158.1.21 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 1.75 MBytes 14.5 Mbits/sec
[ 7] 0.00-1.01 sec 1.75 MBytes 14.5 Mbits/sec
[ 9] 0.00-1.01 sec 1.75 MBytes 14.5 Mbits/sec
[ 11] 0.00-1.01 sec 1.75 MBytes 14.5 Mbits/sec
[ 13] 0.00-1.01 sec 1.75 MBytes 14.5 Mbits/sec
[ 15] 0.00-1.01 sec 1.75 MBytes 14.5 Mbits/sec
[ 17] 0.00-1.01 sec 1.62 MBytes 13.4 Mbits/sec
[ 19] 0.00-1.01 sec 1.75 MBytes 14.5 Mbits/sec
[ 21] 0.00-1.01 sec 1.62 MBytes 13.4 Mbits/sec
[ 23] 0.00-1.01 sec 1.62 MBytes 13.4 Mbits/sec
[SUM] 0.00-1.01 sec 17.1 MBytes 142 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.01-2.01 sec 1.88 MBytes 15.8 Mbits/sec
[ 7] 1.01-2.01 sec 1.88 MBytes 15.8 Mbits/sec
[ 9] 1.01-2.01 sec 1.62 MBytes 13.7 Mbits/sec
[ 11] 1.01-2.01 sec 1.88 MBytes 15.8 Mbits/sec
[ 13] 1.01-2.01 sec 1.62 MBytes 13.7 Mbits/sec
[ 15] 1.01-2.01 sec 1.75 MBytes 14.7 Mbits/sec
[ 17] 1.01-2.01 sec 1.75 MBytes 14.7 Mbits/sec
[ 19] 1.01-2.01 sec 1.75 MBytes 14.7 Mbits/sec
[ 21] 1.01-2.01 sec 1.75 MBytes 14.7 Mbits/sec
[ 23] 1.01-2.01 sec 1.75 MBytes 14.7 Mbits/sec
[SUM] 1.01-2.01 sec 17.6 MBytes 148 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 18.1 MBytes 15.2 Mbits/sec sender
[ 5] 0.00-10.04 sec 18.0 MBytes 15.1 Mbits/sec receiver
[ 7] 0.00-10.01 sec 18.1 MBytes 15.2 Mbits/sec sender
[ 7] 0.00-10.04 sec 18.0 MBytes 15.1 Mbits/sec receiver
[ 9] 0.00-10.01 sec 17.1 MBytes 14.4 Mbits/sec sender
[ 9] 0.00-10.04 sec 17.0 MBytes 14.2 Mbits/sec receiver
[ 11] 0.00-10.01 sec 18.1 MBytes 15.2 Mbits/sec sender
[ 11] 0.00-10.04 sec 18.0 MBytes 15.0 Mbits/sec receiver
[ 13] 0.00-10.01 sec 17.0 MBytes 14.3 Mbits/sec sender
[ 13] 0.00-10.04 sec 16.9 MBytes 14.1 Mbits/sec receiver
[ 15] 0.00-10.01 sec 17.2 MBytes 14.5 Mbits/sec sender
[ 15] 0.00-10.04 sec 17.1 MBytes 14.3 Mbits/sec receiver
[ 17] 0.00-10.01 sec 16.8 MBytes 14.0 Mbits/sec sender
[ 17] 0.00-10.04 sec 16.6 MBytes 13.8 Mbits/sec receiver
[ 19] 0.00-10.01 sec 17.5 MBytes 14.7 Mbits/sec sender
[ 19] 0.00-10.04 sec 17.4 MBytes 14.5 Mbits/sec receiver
[ 21] 0.00-10.01 sec 17.1 MBytes 14.4 Mbits/sec sender
[ 21] 0.00-10.04 sec 17.0 MBytes 14.2 Mbits/sec receiver
[ 23] 0.00-10.01 sec 17.4 MBytes 14.6 Mbits/sec sender
[ 23] 0.00-10.04 sec 17.2 MBytes 14.4 Mbits/sec receiver
[SUM] 0.00-10.01 sec 174 MBytes 146 Mbits/sec sender
[SUM] 0.00-10.04 sec 173 MBytes 145 Mbits/sec receiver
Does anybody have any idea ?
I havent setup any firewall rules except for the most basic ones ...
First of all, this is not routing performance but throughput performance.
Second what is your CPU utilization when you see performance degradation?
Are you running any IPS or ZenArmor?
Regards,
S.
Hey! I have one of those: HDPlex H10 (https://www.tailbone.net/projects/H10/index.html).
I don't have much specific to say about it. OPNsense is reasonably well optimized by default, but you could try further optimizations (lots of threads on that here). The CPU is a fixed 2.9GHz dual-core Haswell, so there's a limit to what you can squeeze out of it.
Quote from: Seimus on March 27, 2025, 03:44:41 PMFirst of all, this is not routing performance but throughput performance.
Second what is your CPU utilization when you see performance degradation?
Are you running any IPS or ZenArmor?
Regards,
S.
The Maximum I´ve seen ( even while running Iperf ) was around 0,3 in the Dashboard.
I dont use any IPS and ZenArmor - at least not that I know of ( relativly new to OPNsense, I´ve used pfSense and OpenWRT before ).
If this OPNsense out of the box than you have no IPS or ZenArmor preinstalled. Basically you have just lean Packet Firewall.
That 0,3 I believe you mean load? And whats the utilization?
Also what NICs has the Sophos, intel or realtek?
Regards,
S.
Quote from: Seimus on March 27, 2025, 04:30:31 PMIf this OPNsense out of the box than you have no IPS or ZenArmor preinstalled. Basically you have just lean Packet Firewall.
That 0,3 I believe you mean load? And whats the utilization?
Also what NICs has the Sophos, intel or realtek?
Regards,
S.
The NICs are Intel, they show up as IGB0-5 ( I use 0 for LAN, 5 for WAN ).
Regarding the utilization - i only have the dashboard widget as reference - any other way to get the info ?
Quote from: Kamikaze on March 27, 2025, 05:11:28 PM[...]
Regarding the utilization - i only have the dashboard widget as reference - any other way to get the info ?
One would be "top", from a console/terminal. But the widget (as of v25 or so) has options to display System/User/Interrupt/Total, so it's not a bad general view. It just won't report on specific processes.
Quote from: pfry on March 27, 2025, 10:11:56 PMQuote from: Kamikaze on March 27, 2025, 05:11:28 PM[...]
Regarding the utilization - i only have the dashboard widget as reference - any other way to get the info ?
One would be "top", from a console/terminal. But the widget (as of v25 or so) has options to display System/User/Interrupt/Total, so it's not a bad general view. It just won't report on specific processes.
during a Public IPerf => Sophos test:
Total hat a spike at 35
Interrupt to 9
User 22
System 17
Update:
I installed my old OpenWRT router that was running fine on our 1000/50 cable.
Absolutely same results and that's with completly different hardware even with client directly connected to the router and even different clients.
The moment the traffic goes to the LAN and not the external IP of the router and inet connection becomes really slow.
Is that throttling from the ISP ?
SOLVED !
We ran multiple test over the weekend again, even it turned out that the client we testet on were the problem.
When we used a linux live stick the performance was as expected and from there on the issue were found with the tcp autotuning.
after running these commands the bandwith exploded:
netsh int tcp set global autotuninglevel=normal
netsh int tcp set global rss=enabled