OPNsense Forum
Archive => 20.1 Legacy Series => Topic started by: TheSmoker on June 30, 2020, 04:25:09 pm
-
Hello,
I am trying to deploy a new gateway for a small but demand home network: mine. ;D
For the hardware I am using a supermicro server : https://www.supermicro.com/en/products/system/1u/5019/SYS-5019D-FN8TP.cfm
Which should be quite sufficient in regards to hardware.
Up until now I've installed OpnSense and updated it to the latest version:
OPNsense 20.1.7-amd64
FreeBSD 11.2-RELEASE-p20-HBSD
OpenSSL 1.1.1g 21 Apr 2020
In rest nothing else installed no pf rules and no other services running.
I have configured one of the SFP+ ports with a Twinax/DAC cable, as a wan interface and add 1 single pf rule which permits all traffic. The server is connected to a Ubiquiti XG 6 switch. No VLANS and no other fancy stuff. This is only used for testing.
My computer is a 9900K with an Aquantia AQC107 network card linked to the same switch with CAT6 cable.
I've started iperf3 on the opnsense and i've used my computer as client. The results are as follows:
iperf-3.1.3-win64>iperf3.exe -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[ 4] local 192.168.1.99 port 54029 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 175 MBytes 1.47 Gbits/sec
[ 4] 1.00-2.00 sec 194 MBytes 1.63 Gbits/sec
[ 4] 2.00-3.00 sec 200 MBytes 1.68 Gbits/sec
[ 4] 3.00-4.00 sec 194 MBytes 1.62 Gbits/sec
[ 4] 4.00-5.00 sec 200 MBytes 1.68 Gbits/sec
[ 4] 5.00-6.00 sec 193 MBytes 1.62 Gbits/sec
[ 4] 6.00-7.00 sec 196 MBytes 1.64 Gbits/sec
[ 4] 7.00-8.00 sec 196 MBytes 1.64 Gbits/sec
[ 4] 8.00-9.00 sec 193 MBytes 1.62 Gbits/sec
[ 4] 9.00-10.00 sec 192 MBytes 1.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.89 GBytes 1.62 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.89 GBytes 1.62 Gbits/sec receiver
iperf Done.
iperf-3.1.3-win64>iperf3.exe -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[ 4] local 192.168.1.99 port 54050 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 109 MBytes 913 Mbits/sec
[ 4] 1.00-2.00 sec 113 MBytes 949 Mbits/sec
[ 4] 2.00-3.00 sec 113 MBytes 949 Mbits/sec
[ 4] 3.00-4.00 sec 113 MBytes 949 Mbits/sec
[ 4] 4.00-5.00 sec 113 MBytes 948 Mbits/sec
[ 4] 5.00-6.00 sec 113 MBytes 949 Mbits/sec
[ 4] 6.00-7.00 sec 113 MBytes 949 Mbits/sec
[ 4] 7.00-8.00 sec 113 MBytes 949 Mbits/sec
[ 4] 8.00-9.00 sec 113 MBytes 949 Mbits/sec
[ 4] 9.00-10.00 sec 113 MBytes 946 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec 2 sender
[ 4] 0.00-10.00 sec 1.10 GBytes 945 Mbits/sec receiver
iperf Done.
:'( :'( :'( :'(
I've switched the DAC cable to the other port (ixl2). Link up. Then without configuring anything else in GUI i run dhclient for that network card and re-ran the iperf3 tests. And the results:
iperf3.exe -c 192.168.1.139
Connecting to host 192.168.1.139, port 5201
[ 4] local 192.168.1.99 port 52943 connected to 192.168.1.139 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 311 MBytes 2.61 Gbits/sec
[ 4] 1.00-2.00 sec 326 MBytes 2.74 Gbits/sec
[ 4] 2.00-3.00 sec 343 MBytes 2.88 Gbits/sec
[ 4] 3.00-4.00 sec 330 MBytes 2.77 Gbits/sec
[ 4] 4.00-5.00 sec 330 MBytes 2.77 Gbits/sec
[ 4] 5.00-6.00 sec 335 MBytes 2.81 Gbits/sec
[ 4] 6.00-7.00 sec 342 MBytes 2.87 Gbits/sec
[ 4] 7.00-8.00 sec 354 MBytes 2.97 Gbits/sec
[ 4] 8.00-9.00 sec 349 MBytes 2.93 Gbits/sec
[ 4] 9.00-10.00 sec 355 MBytes 2.98 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.30 GBytes 2.83 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.30 GBytes 2.83 Gbits/sec receiver
iperf Done.
iperf-3.1.3-win64>iperf3.exe -c 192.168.1.139 -R
Connecting to host 192.168.1.139, port 5201
Reverse mode, remote host 192.168.1.139 is sending
[ 4] local 192.168.1.99 port 52970 connected to 192.168.1.139 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 975 MBytes 8.18 Gbits/sec
[ 4] 1.00-2.00 sec 1.06 GBytes 9.13 Gbits/sec
[ 4] 2.00-3.00 sec 1.06 GBytes 9.06 Gbits/sec
[ 4] 3.00-4.00 sec 1.06 GBytes 9.08 Gbits/sec
[ 4] 4.00-5.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 5.00-6.00 sec 1.06 GBytes 9.12 Gbits/sec
[ 4] 6.00-7.00 sec 1.08 GBytes 9.30 Gbits/sec
[ 4] 7.00-8.00 sec 1.04 GBytes 8.91 Gbits/sec
[ 4] 8.00-9.00 sec 1.10 GBytes 9.44 Gbits/sec
[ 4] 9.00-10.00 sec 1.09 GBytes 9.32 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.6 GBytes 9.10 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 10.6 GBytes 9.10 Gbits/sec receiver
iperf Done.
Something is really wrong here....
Any ideeas?
Cheers!
-
I've also done the following test. Assigned no interfaces. Reboot OPNSense.
Open shell on server and assign IPs to both 10G interfaces:
Then, start iperf3 testing:
http://iperf3.exe -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[ 4] local 192.168.1.99 port 55187 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 256 MBytes 2.15 Gbits/sec
[ 4] 1.00-2.00 sec 264 MBytes 2.22 Gbits/sec
[ 4] 2.00-3.00 sec 299 MBytes 2.50 Gbits/sec
[ 4] 3.00-4.00 sec 60.2 MBytes 505 Mbits/sec
[ 4] 4.00-5.00 sec 197 MBytes 1.65 Gbits/sec
[ 4] 5.00-6.00 sec 228 MBytes 1.91 Gbits/sec
[ 4] 6.00-7.00 sec 225 MBytes 1.89 Gbits/sec
[ 4] 7.00-8.00 sec 245 MBytes 2.05 Gbits/sec
[ 4] 8.00-9.00 sec 306 MBytes 2.57 Gbits/sec
[ 4] 9.00-10.00 sec 293 MBytes 2.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 2.32 GBytes 1.99 Gbits/sec sender
[ 4] 0.00-10.00 sec 2.32 GBytes 1.99 Gbits/sec receiver
iperf Done.
iperf-3.1.3-win64>iperf3.exe -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[ 4] local 192.168.1.99 port 55236 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1004 MBytes 8.43 Gbits/sec
[ 4] 1.00-2.00 sec 1.05 GBytes 9.06 Gbits/sec
[ 4] 2.00-3.00 sec 1.06 GBytes 9.08 Gbits/sec
[ 4] 3.00-4.00 sec 1.08 GBytes 9.24 Gbits/sec
[ 4] 4.00-5.00 sec 1.09 GBytes 9.34 Gbits/sec
[ 4] 5.00-6.00 sec 1.03 GBytes 8.81 Gbits/sec
[ 4] 6.00-7.00 sec 1.09 GBytes 9.34 Gbits/sec
[ 4] 7.00-8.00 sec 1.06 GBytes 9.11 Gbits/sec
[ 4] 8.00-9.00 sec 1.05 GBytes 9.02 Gbits/sec
[ 4] 9.00-10.00 sec 1.02 GBytes 8.79 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.5 GBytes 9.02 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 10.5 GBytes 9.02 Gbits/sec receiver
iperf Done.
This was the previously WAN assigned interface. What's going on?
Same test using my computer as the iperf3 server:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.138, port 39562
[ 5] local 192.168.1.99 port 5201 connected to 192.168.1.138 port 39563
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 189 MBytes 1.58 Gbits/sec
[ 5] 1.00-2.00 sec 999 MBytes 8.38 Gbits/sec
[ 5] 2.00-3.00 sec 980 MBytes 8.22 Gbits/sec
[ 5] 3.00-4.00 sec 1.06 GBytes 9.11 Gbits/sec
[ 5] 4.00-5.00 sec 1.02 GBytes 8.74 Gbits/sec
[ 5] 5.00-6.00 sec 944 MBytes 7.92 Gbits/sec
[ 5] 6.00-7.00 sec 1.08 GBytes 9.32 Gbits/sec
[ 5] 7.00-8.00 sec 1.09 GBytes 9.36 Gbits/sec
[ 5] 8.00-9.00 sec 1.09 GBytes 9.33 Gbits/sec
[ 5] 9.00-10.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 5] 10.00-10.76 sec 834 MBytes 9.15 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.76 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.76 sec 10.3 GBytes 8.21 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.138, port 39564
[ 5] local 192.168.1.99 port 5201 connected to 192.168.1.138 port 39565
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 5.12 MBytes 43.0 Mbits/sec
[ 5] 1.00-2.00 sec 326 MBytes 2.74 Gbits/sec
[ 5] 2.00-3.00 sec 354 MBytes 2.97 Gbits/sec
[ 5] 3.00-4.00 sec 314 MBytes 2.63 Gbits/sec
[ 5] 4.00-5.00 sec 392 MBytes 3.29 Gbits/sec
[ 5] 5.00-6.00 sec 350 MBytes 2.93 Gbits/sec
[ 5] 6.00-7.00 sec 336 MBytes 2.82 Gbits/sec
[ 5] 7.00-8.00 sec 307 MBytes 2.57 Gbits/sec
[ 5] 8.00-9.00 sec 395 MBytes 3.32 Gbits/sec
[ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 10.00-10.91 sec 316 MBytes 2.91 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.91 sec 3.30 GBytes 2.60 Gbits/sec sender
[ 5] 0.00-10.91 sec 0.00 Bytes 0.00 bits/sec receiver
Cheers!
-
1 day and 1 reboot later. nothing changed from yesterday with today. what the hell is going on?
iperf-3.1.3-win64>iperf3.exe -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[ 4] local 192.168.1.99 port 50593 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 162 MBytes 1.36 Gbits/sec
[ 4] 1.00-2.00 sec 197 MBytes 1.65 Gbits/sec
[ 4] 2.00-3.00 sec 201 MBytes 1.69 Gbits/sec
[ 4] 3.00-4.00 sec 198 MBytes 1.66 Gbits/sec
[ 4] 4.00-5.00 sec 188 MBytes 1.58 Gbits/sec
[ 4] 5.00-6.00 sec 190 MBytes 1.60 Gbits/sec
[ 4] 6.00-7.00 sec 191 MBytes 1.60 Gbits/sec
[ 4] 7.00-8.00 sec 192 MBytes 1.61 Gbits/sec
[ 4] 8.00-9.00 sec 192 MBytes 1.61 Gbits/sec
[ 4] 9.00-10.00 sec 198 MBytes 1.66 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.86 GBytes 1.60 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.86 GBytes 1.60 Gbits/sec receiver
iperf Done.
iperf3.exe -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[ 4] local 192.168.1.99 port 50619 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 81.8 MBytes 687 Mbits/sec
[ 4] 1.00-2.00 sec 78.6 MBytes 660 Mbits/sec
[ 4] 2.00-3.00 sec 90.4 MBytes 758 Mbits/sec
[ 4] 3.00-4.00 sec 90.8 MBytes 762 Mbits/sec
[ 4] 4.00-5.00 sec 99.4 MBytes 834 Mbits/sec
[ 4] 5.00-6.00 sec 92.4 MBytes 775 Mbits/sec
[ 4] 6.00-7.00 sec 85.7 MBytes 719 Mbits/sec
[ 4] 7.00-8.00 sec 88.0 MBytes 738 Mbits/sec
[ 4] 8.00-9.00 sec 87.1 MBytes 730 Mbits/sec
[ 4] 9.00-10.00 sec 89.2 MBytes 749 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 884 MBytes 741 Mbits/sec 203 sender
[ 4] 0.00-10.00 sec 884 MBytes 741 Mbits/sec receiver
iperf Done.
-
I may be completely wrong, but I remember a conversation about iperf over on the ubiquiti forums a while ago. Someone had run a similar setup and experts (presumably) chimed in saying that sort of test isn't valid, you need to run iperf through the router to test routing performance, not test how fast the router can run the iperf server itself. That might be a test worth doing if you haven't yet...just a thought.
-
You are partially right. I wasnt expecting wire speed performance from a singe threaded application which runs on a cpu @ 2.0-2.3Ghz. But still the variance if way to big between different ports on same physical quad port 10G interface, not to raise some eyebrows.
Also I cannot explain the variance between reboots. This is screaming unreliable... at least for me. I will do some further testing involving also other OSes including vanilla FreeBSD 11 & 12.
I have to admit that I was expecting a nicer OOTB experience without hassles. With hassles and lot of work I know I can deploy and tune FBSD for almost wire speed 10G.
And no, the UBNT switch has nothing to do with it. I've used the same test to a freenas box with way better results....
iperf3 -c 192.168.1.254
Connecting to host 192.168.1.254, port 5201
[ 4] local 192.168.1.99 port 59782 connected to 192.168.1.254 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 454 MBytes 3.81 Gbits/sec
[ 4] 1.00-2.00 sec 478 MBytes 4.01 Gbits/sec
[ 4] 2.00-3.00 sec 493 MBytes 4.13 Gbits/sec
[ 4] 3.00-4.00 sec 497 MBytes 4.17 Gbits/sec
[ 4] 4.00-5.00 sec 499 MBytes 4.18 Gbits/sec
[ 4] 5.00-6.00 sec 495 MBytes 4.15 Gbits/sec
[ 4] 6.00-7.00 sec 499 MBytes 4.19 Gbits/sec
[ 4] 7.00-8.00 sec 497 MBytes 4.17 Gbits/sec
[ 4] 8.00-9.00 sec 500 MBytes 4.19 Gbits/sec
[ 4] 9.00-10.00 sec 498 MBytes 4.18 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.79 GBytes 4.12 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.79 GBytes 4.12 Gbits/sec receiver
iperf Done.
iperf-3.1.3-win64>iperf3 -c 192.168.1.254 -R
Connecting to host 192.168.1.254, port 5201
Reverse mode, remote host 192.168.1.254 is sending
[ 4] local 192.168.1.99 port 59794 connected to 192.168.1.254 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 813 MBytes 6.82 Gbits/sec
[ 4] 1.00-2.00 sec 858 MBytes 7.20 Gbits/sec
[ 4] 2.00-3.00 sec 859 MBytes 7.21 Gbits/sec
[ 4] 3.00-4.00 sec 859 MBytes 7.21 Gbits/sec
[ 4] 4.00-5.00 sec 864 MBytes 7.25 Gbits/sec
[ 4] 5.00-6.00 sec 862 MBytes 7.23 Gbits/sec
[ 4] 6.00-7.00 sec 860 MBytes 7.21 Gbits/sec
[ 4] 7.00-8.00 sec 860 MBytes 7.22 Gbits/sec
[ 4] 8.00-9.00 sec 864 MBytes 7.25 Gbits/sec
[ 4] 9.00-10.00 sec 865 MBytes 7.25 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 8.36 GBytes 7.18 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 8.36 GBytes 7.18 Gbits/sec receiver
iperf Done.
-
Same hardware & switching config with pfSense 2.4.5-RELEASE-P1:
iperf-3.1.3-win64>iperf3 -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[ 4] local 192.168.1.99 port 59913 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 355 MBytes 2.97 Gbits/sec
[ 4] 1.00-2.00 sec 368 MBytes 3.09 Gbits/sec
[ 4] 2.00-3.00 sec 376 MBytes 3.16 Gbits/sec
[ 4] 3.00-4.00 sec 379 MBytes 3.18 Gbits/sec
[ 4] 4.00-5.00 sec 374 MBytes 3.14 Gbits/sec
[ 4] 5.00-6.00 sec 376 MBytes 3.15 Gbits/sec
[ 4] 6.00-7.00 sec 373 MBytes 3.13 Gbits/sec
[ 4] 7.00-8.00 sec 376 MBytes 3.16 Gbits/sec
[ 4] 8.00-9.00 sec 371 MBytes 3.11 Gbits/sec
[ 4] 9.00-10.00 sec 372 MBytes 3.12 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.63 GBytes 3.12 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.63 GBytes 3.12 Gbits/sec receiver
iperf Done.
iperf-3.1.3-win64>iperf3 -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[ 4] local 192.168.1.99 port 59954 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 559 MBytes 4.69 Gbits/sec
[ 4] 1.00-2.00 sec 580 MBytes 4.87 Gbits/sec
[ 4] 2.00-3.00 sec 583 MBytes 4.90 Gbits/sec
[ 4] 3.00-4.00 sec 583 MBytes 4.89 Gbits/sec
[ 4] 4.00-5.00 sec 582 MBytes 4.88 Gbits/sec
[ 4] 5.00-6.00 sec 581 MBytes 4.88 Gbits/sec
[ 4] 6.00-7.00 sec 583 MBytes 4.89 Gbits/sec
[ 4] 7.00-8.00 sec 581 MBytes 4.88 Gbits/sec
[ 4] 8.00-9.00 sec 580 MBytes 4.87 Gbits/sec
[ 4] 9.00-10.00 sec 581 MBytes 4.87 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 5.66 GBytes 4.86 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 5.66 GBytes 4.86 Gbits/sec receiver
iperf Done.
-
Same thing with FreeBSD 12.1-RELEASE installed:
iperf-3.1.3-win64>iperf3 -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[ 4] local 192.168.1.99 port 62735 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 446 MBytes 3.74 Gbits/sec
[ 4] 1.00-2.00 sec 454 MBytes 3.81 Gbits/sec
[ 4] 2.00-3.00 sec 456 MBytes 3.82 Gbits/sec
[ 4] 3.00-4.00 sec 459 MBytes 3.85 Gbits/sec
[ 4] 4.00-5.00 sec 468 MBytes 3.93 Gbits/sec
[ 4] 5.00-6.00 sec 464 MBytes 3.89 Gbits/sec
[ 4] 6.00-7.00 sec 461 MBytes 3.87 Gbits/sec
[ 4] 7.00-8.00 sec 460 MBytes 3.86 Gbits/sec
[ 4] 8.00-9.00 sec 465 MBytes 3.90 Gbits/sec
[ 4] 9.00-10.00 sec 464 MBytes 3.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 4.49 GBytes 3.86 Gbits/sec sender
[ 4] 0.00-10.00 sec 4.49 GBytes 3.86 Gbits/sec receiver
iperf Done.
iperf3 -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[ 4] local 192.168.1.99 port 62747 connected to 192.168.1.138 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 795 MBytes 6.67 Gbits/sec
[ 4] 1.00-2.00 sec 844 MBytes 7.08 Gbits/sec
[ 4] 2.00-3.00 sec 849 MBytes 7.13 Gbits/sec
[ 4] 3.00-4.00 sec 851 MBytes 7.14 Gbits/sec
[ 4] 4.00-5.00 sec 854 MBytes 7.16 Gbits/sec
[ 4] 5.00-6.00 sec 854 MBytes 7.17 Gbits/sec
[ 4] 6.00-7.00 sec 851 MBytes 7.14 Gbits/sec
[ 4] 7.00-8.00 sec 851 MBytes 7.14 Gbits/sec
[ 4] 8.00-9.00 sec 857 MBytes 7.19 Gbits/sec
[ 4] 9.00-10.00 sec 850 MBytes 7.13 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 8.26 GBytes 7.09 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 8.26 GBytes 7.09 Gbits/sec receiver
iperf Done.
-
I can confirm that 10G experience with opnsense is lacking...
I have a somewhat similar setup to yours: Two opnsense 20.1 Supermicro based firewalls with Intel i3-9100 CPUs and Chelsio T540-CR cards in them.
When I iperf3 between firewalls on intranet interface in same subnet (no rules, no nat, no routing) I come to very similar results as you (~2 Gbits/sec), if I run several streams I can bring it up to 3-4Gbits/sec total but not more.
If on another hand I run out of the box freebsd on the same machines I get 4-5 Gbits/sec on single thread and 7-8 Gbits/sec on several threads.
Opnsense seems to have issues there, I am not sure what the problem is as I had no time to dig deeper and my internet uplink is just 1Gbits/sec anyway.
-
I cant repeat myself often enough. Never ever use iperf in the Firewall itself. A real lab consits of two Firewalls and two clients behind. I did extensive testing .. also with 40g cards and also all kinds of VPNs.
Somewhere on my page I posted the results.
-
I cant repeat myself often enough. Never ever use iperf in the Firewall itself. A real lab consits of two Firewalls and two clients behind. I did extensive testing .. also with 40g cards and also all kinds of VPNs.
Somewhere on my page I posted the results.
I did not had the time, but I have 2 nodes with 8600K cpus and 10G Supermicro/Intel NICs and I will use them as packet generator and packet sink to test routing/nat performance and I will post back.
I can confirm right now that disabling hyperthreading and modifying interrupt harvest mask to 351 improves performance a bit. This kind of basic tuning should be provided OOTB in a router product ... just my 2 cents here ...
-
Here are some results from my 10G lab.
I also tested with untuned OPN and 40G cards from Chelsio which gave me around 22Gbit without NAT.
If you don't need netmap support you can enable hardware offloading which should bring you to wirespeed (if CPU permits)
-
Here are some results from my 10G lab.
I also tested with untuned OPN and 40G cards from Chelsio which gave me around 22Gbit without NAT.
If you don't need netmap support you can enable hardware offloading which should bring you to wirespeed (if CPU permits)
You wanted to give a link to something?
-
Sorry man, my mistake :)
https://www.routerperformance.net/routers/nexcom-nsa/fujitsu-rx1330/
https://www.routerperformance.net/routers/nexcom-nsa/thomas-krenn-ri1102d/
https://www.routerperformance.net/comparing-opnsense-vpn-performance/
Currently I'm refreshing my lab with newer hardware but 10G only .. if you want me to test something, just ping me here :) (need to wait for the apprentice setting up the clients behind gateways)
-
Ok, I got fresh hardware and first test results .. much better than with the old test equipment:
https://www.routerperformance.net/opnsense/opnsense-performance-20-1-8/
I'll add results for IPsec, Wireguard and IPS over the weekend 8)
-
Yet another point of reference; I use much older hardware (Dell T320 w/ Xeon E5-2450 v2, Mellanox ConnectX-3 EN based LAN cards, ESXi6.5 w/ numerous VMs - one of which happens to be OPNsense). In summary, not so ideal from an OPNsense CPU/install point of view.
With hardware offload disabled (IPS running only on WAN), routing via OPNsense (LAN subnets) I am limited to around 1400Mbit.
With offload enabled (IPS off), performance is closer to 9400Mbit.
My guess is if you are limited by your hardware, you need to choose either security or speed - depending on your needs. Default appears to be security (and stability in the case of some cards offload capability) - which I think is the correct way to lean in this case.