iperf3 speed difference with 10G networking

[TheSmoker]

Hello,

I am trying to deploy a new gateway for a small but demand home network: mine. 

For the hardware I am using a supermicro server : https://www.supermicro.com/en/products/system/1u/5019/SYS-5019D-FN8TP.cfm

Which should be quite sufficient in regards to hardware.
Up until now I've installed OpnSense and updated it to the latest version:
OPNsense 20.1.7-amd64
FreeBSD 11.2-RELEASE-p20-HBSD
OpenSSL 1.1.1g 21 Apr 2020

In rest nothing else installed no pf rules and no other services running.

I have configured one of the SFP+ ports with a Twinax/DAC cable, as a wan interface and add 1 single pf rule which permits all traffic. The server is connected to a Ubiquiti XG 6 switch. No VLANS and no other fancy stuff. This is only used for testing.

My computer is a 9900K with an Aquantia AQC107 network card linked to the same switch with CAT6 cable.

I've started iperf3 on the opnsense and i've used my computer as client. The results are as follows:

Code: [Select]

iperf-3.1.3-win64>iperf3.exe -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[  4] local 192.168.1.99 port 54029 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   175 MBytes  1.47 Gbits/sec
[  4]   1.00-2.00   sec   194 MBytes  1.63 Gbits/sec
[  4]   2.00-3.00   sec   200 MBytes  1.68 Gbits/sec
[  4]   3.00-4.00   sec   194 MBytes  1.62 Gbits/sec
[  4]   4.00-5.00   sec   200 MBytes  1.68 Gbits/sec
[  4]   5.00-6.00   sec   193 MBytes  1.62 Gbits/sec
[  4]   6.00-7.00   sec   196 MBytes  1.64 Gbits/sec
[  4]   7.00-8.00   sec   196 MBytes  1.64 Gbits/sec
[  4]   8.00-9.00   sec   193 MBytes  1.62 Gbits/sec
[  4]   9.00-10.00  sec   192 MBytes  1.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.89 GBytes  1.62 Gbits/sec                  sender
[  4]   0.00-10.00  sec  1.89 GBytes  1.62 Gbits/sec                  receiver

iperf Done.

iperf-3.1.3-win64>iperf3.exe -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[  4] local 192.168.1.99 port 54050 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   109 MBytes   913 Mbits/sec
[  4]   1.00-2.00   sec   113 MBytes   949 Mbits/sec
[  4]   2.00-3.00   sec   113 MBytes   949 Mbits/sec
[  4]   3.00-4.00   sec   113 MBytes   949 Mbits/sec
[  4]   4.00-5.00   sec   113 MBytes   948 Mbits/sec
[  4]   5.00-6.00   sec   113 MBytes   949 Mbits/sec
[  4]   6.00-7.00   sec   113 MBytes   949 Mbits/sec
[  4]   7.00-8.00   sec   113 MBytes   949 Mbits/sec
[  4]   8.00-9.00   sec   113 MBytes   949 Mbits/sec
[  4]   9.00-10.00  sec   113 MBytes   946 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.10 GBytes   945 Mbits/sec    2             sender
[  4]   0.00-10.00  sec  1.10 GBytes   945 Mbits/sec                  receiver

iperf Done.
 

I've switched the DAC cable to the other port (ixl2). Link up. Then without configuring anything else in GUI i run dhclient for that network card and re-ran the iperf3 tests. And the results:

Code: [Select]

iperf3.exe -c 192.168.1.139
Connecting to host 192.168.1.139, port 5201
[  4] local 192.168.1.99 port 52943 connected to 192.168.1.139 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   311 MBytes  2.61 Gbits/sec
[  4]   1.00-2.00   sec   326 MBytes  2.74 Gbits/sec
[  4]   2.00-3.00   sec   343 MBytes  2.88 Gbits/sec
[  4]   3.00-4.00   sec   330 MBytes  2.77 Gbits/sec
[  4]   4.00-5.00   sec   330 MBytes  2.77 Gbits/sec
[  4]   5.00-6.00   sec   335 MBytes  2.81 Gbits/sec
[  4]   6.00-7.00   sec   342 MBytes  2.87 Gbits/sec
[  4]   7.00-8.00   sec   354 MBytes  2.97 Gbits/sec
[  4]   8.00-9.00   sec   349 MBytes  2.93 Gbits/sec
[  4]   9.00-10.00  sec   355 MBytes  2.98 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.30 GBytes  2.83 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.30 GBytes  2.83 Gbits/sec                  receiver

iperf Done.

iperf-3.1.3-win64>iperf3.exe -c 192.168.1.139 -R
Connecting to host 192.168.1.139, port 5201
Reverse mode, remote host 192.168.1.139 is sending
[  4] local 192.168.1.99 port 52970 connected to 192.168.1.139 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   975 MBytes  8.18 Gbits/sec
[  4]   1.00-2.00   sec  1.06 GBytes  9.13 Gbits/sec
[  4]   2.00-3.00   sec  1.06 GBytes  9.06 Gbits/sec
[  4]   3.00-4.00   sec  1.06 GBytes  9.08 Gbits/sec
[  4]   4.00-5.00   sec  1.10 GBytes  9.41 Gbits/sec
[  4]   5.00-6.00   sec  1.06 GBytes  9.12 Gbits/sec
[  4]   6.00-7.00   sec  1.08 GBytes  9.30 Gbits/sec
[  4]   7.00-8.00   sec  1.04 GBytes  8.91 Gbits/sec
[  4]   8.00-9.00   sec  1.10 GBytes  9.44 Gbits/sec
[  4]   9.00-10.00  sec  1.09 GBytes  9.32 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.6 GBytes  9.10 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  10.6 GBytes  9.10 Gbits/sec                  receiver

iperf Done.

Something is really wrong here....

Any ideeas?

Cheers!

[TheSmoker]

I've also done the following test. Assigned no interfaces. Reboot OPNSense.
Open shell on server and assign IPs to both 10G interfaces:

Then, start iperf3 testing:

Code: [Select]

http://iperf3.exe -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[  4] local 192.168.1.99 port 55187 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   256 MBytes  2.15 Gbits/sec
[  4]   1.00-2.00   sec   264 MBytes  2.22 Gbits/sec
[  4]   2.00-3.00   sec   299 MBytes  2.50 Gbits/sec
[  4]   3.00-4.00   sec  60.2 MBytes   505 Mbits/sec
[  4]   4.00-5.00   sec   197 MBytes  1.65 Gbits/sec
[  4]   5.00-6.00   sec   228 MBytes  1.91 Gbits/sec
[  4]   6.00-7.00   sec   225 MBytes  1.89 Gbits/sec
[  4]   7.00-8.00   sec   245 MBytes  2.05 Gbits/sec
[  4]   8.00-9.00   sec   306 MBytes  2.57 Gbits/sec
[  4]   9.00-10.00  sec   293 MBytes  2.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.32 GBytes  1.99 Gbits/sec                  sender
[  4]   0.00-10.00  sec  2.32 GBytes  1.99 Gbits/sec                  receiver

iperf Done.

iperf-3.1.3-win64>iperf3.exe -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[  4] local 192.168.1.99 port 55236 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1004 MBytes  8.43 Gbits/sec
[  4]   1.00-2.00   sec  1.05 GBytes  9.06 Gbits/sec
[  4]   2.00-3.00   sec  1.06 GBytes  9.08 Gbits/sec
[  4]   3.00-4.00   sec  1.08 GBytes  9.24 Gbits/sec
[  4]   4.00-5.00   sec  1.09 GBytes  9.34 Gbits/sec
[  4]   5.00-6.00   sec  1.03 GBytes  8.81 Gbits/sec
[  4]   6.00-7.00   sec  1.09 GBytes  9.34 Gbits/sec
[  4]   7.00-8.00   sec  1.06 GBytes  9.11 Gbits/sec
[  4]   8.00-9.00   sec  1.05 GBytes  9.02 Gbits/sec
[  4]   9.00-10.00  sec  1.02 GBytes  8.79 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.5 GBytes  9.02 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  10.5 GBytes  9.02 Gbits/sec                  receiver

iperf Done.
This was the previously WAN assigned interface. What's going on?

Same test using my computer as the iperf3 server:

Code: [Select]

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.138, port 39562
[  5] local 192.168.1.99 port 5201 connected to 192.168.1.138 port 39563
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   189 MBytes  1.58 Gbits/sec
[  5]   1.00-2.00   sec   999 MBytes  8.38 Gbits/sec
[  5]   2.00-3.00   sec   980 MBytes  8.22 Gbits/sec
[  5]   3.00-4.00   sec  1.06 GBytes  9.11 Gbits/sec
[  5]   4.00-5.00   sec  1.02 GBytes  8.74 Gbits/sec
[  5]   5.00-6.00   sec   944 MBytes  7.92 Gbits/sec
[  5]   6.00-7.00   sec  1.08 GBytes  9.32 Gbits/sec
[  5]   7.00-8.00   sec  1.09 GBytes  9.36 Gbits/sec
[  5]   8.00-9.00   sec  1.09 GBytes  9.33 Gbits/sec
[  5]   9.00-10.00  sec  1.09 GBytes  9.37 Gbits/sec
[  5]  10.00-10.76  sec   834 MBytes  9.15 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.76  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.76  sec  10.3 GBytes  8.21 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.138, port 39564
[  5] local 192.168.1.99 port 5201 connected to 192.168.1.138 port 39565
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  5.12 MBytes  43.0 Mbits/sec
[  5]   1.00-2.00   sec   326 MBytes  2.74 Gbits/sec
[  5]   2.00-3.00   sec   354 MBytes  2.97 Gbits/sec
[  5]   3.00-4.00   sec   314 MBytes  2.63 Gbits/sec
[  5]   4.00-5.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   5.00-6.00   sec   350 MBytes  2.93 Gbits/sec
[  5]   6.00-7.00   sec   336 MBytes  2.82 Gbits/sec
[  5]   7.00-8.00   sec   307 MBytes  2.57 Gbits/sec
[  5]   8.00-9.00   sec   395 MBytes  3.32 Gbits/sec
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec
[  5]  10.00-10.91  sec   316 MBytes  2.91 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.91  sec  3.30 GBytes  2.60 Gbits/sec                  sender
[  5]   0.00-10.91  sec  0.00 Bytes  0.00 bits/sec                  receiver
Cheers!

[TheSmoker]

1 day and 1 reboot later. nothing changed from yesterday with today. what the hell is going on?

Code: [Select]

iperf-3.1.3-win64>iperf3.exe -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[  4] local 192.168.1.99 port 50593 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   162 MBytes  1.36 Gbits/sec
[  4]   1.00-2.00   sec   197 MBytes  1.65 Gbits/sec
[  4]   2.00-3.00   sec   201 MBytes  1.69 Gbits/sec
[  4]   3.00-4.00   sec   198 MBytes  1.66 Gbits/sec
[  4]   4.00-5.00   sec   188 MBytes  1.58 Gbits/sec
[  4]   5.00-6.00   sec   190 MBytes  1.60 Gbits/sec
[  4]   6.00-7.00   sec   191 MBytes  1.60 Gbits/sec
[  4]   7.00-8.00   sec   192 MBytes  1.61 Gbits/sec
[  4]   8.00-9.00   sec   192 MBytes  1.61 Gbits/sec
[  4]   9.00-10.00  sec   198 MBytes  1.66 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.86 GBytes  1.60 Gbits/sec                  sender
[  4]   0.00-10.00  sec  1.86 GBytes  1.60 Gbits/sec                  receiver

iperf Done.

iperf3.exe -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[  4] local 192.168.1.99 port 50619 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  81.8 MBytes   687 Mbits/sec
[  4]   1.00-2.00   sec  78.6 MBytes   660 Mbits/sec
[  4]   2.00-3.00   sec  90.4 MBytes   758 Mbits/sec
[  4]   3.00-4.00   sec  90.8 MBytes   762 Mbits/sec
[  4]   4.00-5.00   sec  99.4 MBytes   834 Mbits/sec
[  4]   5.00-6.00   sec  92.4 MBytes   775 Mbits/sec
[  4]   6.00-7.00   sec  85.7 MBytes   719 Mbits/sec
[  4]   7.00-8.00   sec  88.0 MBytes   738 Mbits/sec
[  4]   8.00-9.00   sec  87.1 MBytes   730 Mbits/sec
[  4]   9.00-10.00  sec  89.2 MBytes   749 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   884 MBytes   741 Mbits/sec  203             sender
[  4]   0.00-10.00  sec   884 MBytes   741 Mbits/sec                  receiver

iperf Done.

[gpb]

I may be completely wrong, but I remember a conversation about iperf over on the ubiquiti forums a while ago.  Someone had run a similar setup and experts (presumably) chimed in saying that sort of test isn't valid, you need to run iperf through the router to test routing performance, not test how fast the router can run the iperf server itself.  That might be a test worth doing if you haven't yet...just a thought.

[TheSmoker]

You are partially right. I wasnt expecting wire speed performance from a singe threaded application which runs on a cpu @ 2.0-2.3Ghz. But still the variance if way to big between different ports on same physical quad port 10G interface, not to raise some eyebrows.

Also I cannot explain the variance between reboots. This is screaming unreliable... at least for me. I will do some further testing involving also other OSes including vanilla FreeBSD 11 & 12.

I have to admit that I was expecting a nicer OOTB experience without hassles. With hassles and lot of work I know I can deploy and tune FBSD for almost wire speed 10G.

And no, the UBNT switch has nothing to do with it. I've used the same test to a freenas box with way better results....

Code: [Select]

iperf3 -c 192.168.1.254
Connecting to host 192.168.1.254, port 5201
[  4] local 192.168.1.99 port 59782 connected to 192.168.1.254 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   454 MBytes  3.81 Gbits/sec
[  4]   1.00-2.00   sec   478 MBytes  4.01 Gbits/sec
[  4]   2.00-3.00   sec   493 MBytes  4.13 Gbits/sec
[  4]   3.00-4.00   sec   497 MBytes  4.17 Gbits/sec
[  4]   4.00-5.00   sec   499 MBytes  4.18 Gbits/sec
[  4]   5.00-6.00   sec   495 MBytes  4.15 Gbits/sec
[  4]   6.00-7.00   sec   499 MBytes  4.19 Gbits/sec
[  4]   7.00-8.00   sec   497 MBytes  4.17 Gbits/sec
[  4]   8.00-9.00   sec   500 MBytes  4.19 Gbits/sec
[  4]   9.00-10.00  sec   498 MBytes  4.18 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  4.79 GBytes  4.12 Gbits/sec                  sender
[  4]   0.00-10.00  sec  4.79 GBytes  4.12 Gbits/sec                  receiver

iperf Done.

iperf-3.1.3-win64>iperf3 -c 192.168.1.254 -R
Connecting to host 192.168.1.254, port 5201
Reverse mode, remote host 192.168.1.254 is sending
[  4] local 192.168.1.99 port 59794 connected to 192.168.1.254 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   813 MBytes  6.82 Gbits/sec
[  4]   1.00-2.00   sec   858 MBytes  7.20 Gbits/sec
[  4]   2.00-3.00   sec   859 MBytes  7.21 Gbits/sec
[  4]   3.00-4.00   sec   859 MBytes  7.21 Gbits/sec
[  4]   4.00-5.00   sec   864 MBytes  7.25 Gbits/sec
[  4]   5.00-6.00   sec   862 MBytes  7.23 Gbits/sec
[  4]   6.00-7.00   sec   860 MBytes  7.21 Gbits/sec
[  4]   7.00-8.00   sec   860 MBytes  7.22 Gbits/sec
[  4]   8.00-9.00   sec   864 MBytes  7.25 Gbits/sec
[  4]   9.00-10.00  sec   865 MBytes  7.25 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.36 GBytes  7.18 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  8.36 GBytes  7.18 Gbits/sec                  receiver

iperf Done.

[TheSmoker]

Same hardware & switching config with pfSense 2.4.5-RELEASE-P1:

Code: [Select]

iperf-3.1.3-win64>iperf3 -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[  4] local 192.168.1.99 port 59913 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   355 MBytes  2.97 Gbits/sec
[  4]   1.00-2.00   sec   368 MBytes  3.09 Gbits/sec
[  4]   2.00-3.00   sec   376 MBytes  3.16 Gbits/sec
[  4]   3.00-4.00   sec   379 MBytes  3.18 Gbits/sec
[  4]   4.00-5.00   sec   374 MBytes  3.14 Gbits/sec
[  4]   5.00-6.00   sec   376 MBytes  3.15 Gbits/sec
[  4]   6.00-7.00   sec   373 MBytes  3.13 Gbits/sec
[  4]   7.00-8.00   sec   376 MBytes  3.16 Gbits/sec
[  4]   8.00-9.00   sec   371 MBytes  3.11 Gbits/sec
[  4]   9.00-10.00  sec   372 MBytes  3.12 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.63 GBytes  3.12 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.63 GBytes  3.12 Gbits/sec                  receiver

iperf Done.

iperf-3.1.3-win64>iperf3 -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[  4] local 192.168.1.99 port 59954 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   559 MBytes  4.69 Gbits/sec
[  4]   1.00-2.00   sec   580 MBytes  4.87 Gbits/sec
[  4]   2.00-3.00   sec   583 MBytes  4.90 Gbits/sec
[  4]   3.00-4.00   sec   583 MBytes  4.89 Gbits/sec
[  4]   4.00-5.00   sec   582 MBytes  4.88 Gbits/sec
[  4]   5.00-6.00   sec   581 MBytes  4.88 Gbits/sec
[  4]   6.00-7.00   sec   583 MBytes  4.89 Gbits/sec
[  4]   7.00-8.00   sec   581 MBytes  4.88 Gbits/sec
[  4]   8.00-9.00   sec   580 MBytes  4.87 Gbits/sec
[  4]   9.00-10.00  sec   581 MBytes  4.87 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec                  receiver

iperf Done.

[TheSmoker]

Same thing with FreeBSD 12.1-RELEASE installed:

Code: [Select]

iperf-3.1.3-win64>iperf3 -c 192.168.1.138
Connecting to host 192.168.1.138, port 5201
[  4] local 192.168.1.99 port 62735 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   446 MBytes  3.74 Gbits/sec
[  4]   1.00-2.00   sec   454 MBytes  3.81 Gbits/sec
[  4]   2.00-3.00   sec   456 MBytes  3.82 Gbits/sec
[  4]   3.00-4.00   sec   459 MBytes  3.85 Gbits/sec
[  4]   4.00-5.00   sec   468 MBytes  3.93 Gbits/sec
[  4]   5.00-6.00   sec   464 MBytes  3.89 Gbits/sec
[  4]   6.00-7.00   sec   461 MBytes  3.87 Gbits/sec
[  4]   7.00-8.00   sec   460 MBytes  3.86 Gbits/sec
[  4]   8.00-9.00   sec   465 MBytes  3.90 Gbits/sec
[  4]   9.00-10.00  sec   464 MBytes  3.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  4.49 GBytes  3.86 Gbits/sec                  sender
[  4]   0.00-10.00  sec  4.49 GBytes  3.86 Gbits/sec                  receiver

iperf Done.

iperf3 -c 192.168.1.138 -R
Connecting to host 192.168.1.138, port 5201
Reverse mode, remote host 192.168.1.138 is sending
[  4] local 192.168.1.99 port 62747 connected to 192.168.1.138 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   795 MBytes  6.67 Gbits/sec
[  4]   1.00-2.00   sec   844 MBytes  7.08 Gbits/sec
[  4]   2.00-3.00   sec   849 MBytes  7.13 Gbits/sec
[  4]   3.00-4.00   sec   851 MBytes  7.14 Gbits/sec
[  4]   4.00-5.00   sec   854 MBytes  7.16 Gbits/sec
[  4]   5.00-6.00   sec   854 MBytes  7.17 Gbits/sec
[  4]   6.00-7.00   sec   851 MBytes  7.14 Gbits/sec
[  4]   7.00-8.00   sec   851 MBytes  7.14 Gbits/sec
[  4]   8.00-9.00   sec   857 MBytes  7.19 Gbits/sec
[  4]   9.00-10.00  sec   850 MBytes  7.13 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.26 GBytes  7.09 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  8.26 GBytes  7.09 Gbits/sec                  receiver

iperf Done.

[Don.key]

I can confirm that 10G experience with opnsense is lacking...

I have a somewhat similar setup to yours: Two opnsense 20.1 Supermicro based firewalls with Intel i3-9100 CPUs and Chelsio T540-CR cards in them.

When I iperf3 between firewalls on intranet interface in same subnet (no rules, no nat, no routing) I come to very similar results as you (~2 Gbits/sec), if I run several streams I can bring it up to 3-4Gbits/sec total but not more.

If on another hand I run out of the box freebsd on the same machines I get 4-5 Gbits/sec on single thread and 7-8 Gbits/sec on several threads.

Opnsense seems to have issues there, I am not sure what the problem is as I had no time to dig deeper and my internet uplink is just 1Gbits/sec anyway.

[mimugmail]

I cant repeat myself often enough. Never ever use iperf in the Firewall itself. A real lab consits of two Firewalls and two clients behind. I did extensive testing .. also with 40g cards and also all kinds of VPNs.
Somewhere on my page I posted the results.

[TheSmoker]

I cant repeat myself often enough. Never ever use iperf in the Firewall itself. A real lab consits of two Firewalls and two clients behind. I did extensive testing .. also with 40g cards and also all kinds of VPNs.
Somewhere on my page I posted the results.

I did not had the time, but I have 2 nodes with 8600K cpus and 10G Supermicro/Intel NICs and I will use them as packet generator and packet sink to test routing/nat performance and I will post back.

I can confirm right now that disabling hyperthreading and modifying interrupt harvest mask to 351 improves performance a bit. This kind of basic tuning should be provided OOTB in a router product ... just my 2 cents here ...

[mimugmail]

Here are some results from my 10G lab.
I also tested with untuned OPN and 40G cards from Chelsio which gave me around 22Gbit without NAT.

If you don't need netmap support you can enable hardware offloading which should bring you to wirespeed (if CPU permits)

[TheSmoker]

Here are some results from my 10G lab.
I also tested with untuned OPN and 40G cards from Chelsio which gave me around 22Gbit without NAT.

If you don't need netmap support you can enable hardware offloading which should bring you to wirespeed (if CPU permits)

You wanted to give a link to something?

[mimugmail]

Sorry man, my mistake

https://www.routerperformance.net/routers/nexcom-nsa/fujitsu-rx1330/
https://www.routerperformance.net/routers/nexcom-nsa/thomas-krenn-ri1102d/
https://www.routerperformance.net/comparing-opnsense-vpn-performance/


Currently I'm refreshing my lab with newer hardware but 10G only .. if you want me to test something, just ping me here (need to wait for the apprentice setting up the clients behind gateways)

[mimugmail]

Ok, I got fresh hardware and first test results .. much better than with the old test equipment:

https://www.routerperformance.net/opnsense/opnsense-performance-20-1-8/


I'll add results for IPsec, Wireguard and IPS over the weekend 

[youngman]

Yet another point of reference; I use much older hardware (Dell T320 w/ Xeon E5-2450 v2, Mellanox ConnectX-3 EN based LAN cards, ESXi6.5 w/ numerous VMs - one of which happens to be OPNsense). In summary, not so ideal from an OPNsense CPU/install point of view.

With hardware offload disabled (IPS running only on WAN), routing via OPNsense (LAN subnets) I am limited to around 1400Mbit.

With offload enabled (IPS off), performance is closer to 9400Mbit.

My guess is if you are limited by your hardware, you need to choose either security or speed - depending on your needs. Default appears to be security (and stability in the case of some cards offload capability) - which I think is the correct way to lean in this case.