opnsense 10gb low performance higher cpu on iperf and nfs server

[ditrames]

for the better part of a year i have been trying to get opnsense to do 10gb

i have a dell r330 with a Intel(R) Xeon(R) CPU E3-1270 v5 @ 3.60GHz and 8 gb of ram and a mellenox x3 card as the bare metal opnsense box


this is a big issue as this is holding back alot of my devices that need the full throughput

id also like to add when doing iperf tests when on same vlan the cpu useage on one core is 60% ish on the iperf server
when going thru opnsense the cpu is at 100%

why would going through opnsense cause higher cpu usage while delivering slower performance?

edit:
after more testing when going thru opnsense cause the nfs server to peg a core at 100% and peek tranfer at 3-4gbps wtf is going on


i have two machines on two diffrent vlans.

when on the same vlan i get via iperf results:
tcp:

Code: [Select]

Connecting to host 172.16.3.2, port 5201
[  5] local 172.16.3.4 port 53690 connected to 172.16.3.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.39 Gbits/sec  242   1.02 MBytes       
[  5]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec    8   1004 KBytes       
[  5]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec    8    932 KBytes       
[  5]   3.00-4.00   sec  1.09 GBytes  9.37 Gbits/sec    8    980 KBytes       
[  5]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec    8    986 KBytes       
[  5]   5.00-6.00   sec  1.09 GBytes  9.37 Gbits/sec    8    980 KBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.37 Gbits/sec    8    986 KBytes       
[  5]   7.00-8.00   sec  1.09 GBytes  9.38 Gbits/sec    8    928 KBytes       
[  5]   8.00-9.00   sec  1.09 GBytes  9.37 Gbits/sec    8   1.03 MBytes       
[  5]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec    8    923 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec  314             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  receiver


Code: [Select]

Connecting to host 172.16.3.2, port 5201
Reverse mode, remote host 172.16.3.2 is sending
[  5] local 172.16.3.4 port 55870 connected to 172.16.3.2 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.09 GBytes  9.36 Gbits/sec                 
[  5]   1.00-2.00   sec  1.09 GBytes  9.37 Gbits/sec                 
[  5]   2.00-3.00   sec  1.09 GBytes  9.38 Gbits/sec                 
[  5]   3.00-4.00   sec  1.09 GBytes  9.36 Gbits/sec                 
[  5]   4.00-5.00   sec  1.09 GBytes  9.37 Gbits/sec                 
[  5]   5.00-6.00   sec  1.09 GBytes  9.37 Gbits/sec                 
[  5]   6.00-7.00   sec  1.09 GBytes  9.35 Gbits/sec                 
[  5]   7.00-8.00   sec  1.09 GBytes  9.37 Gbits/sec                 
[  5]   8.00-9.00   sec  1.09 GBytes  9.37 Gbits/sec                 
[  5]   9.00-10.00  sec  1.09 GBytes  9.37 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.37 Gbits/sec  128             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.37 Gbits/sec                  receiver
udp:

Code: [Select]

Connecting to host 172.16.3.2, port 5201
[  5] local 172.16.3.4 port 58135 connected to 172.16.3.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   778 MBytes  6.53 Gbits/sec  563560 
[  5]   1.00-2.00   sec   563 MBytes  4.72 Gbits/sec  407500 
[  5]   2.00-3.00   sec   648 MBytes  5.44 Gbits/sec  469190 
[  5]   3.00-4.00   sec   568 MBytes  4.77 Gbits/sec  411550 
[  5]   4.00-5.00   sec   650 MBytes  5.45 Gbits/sec  470620 
[  5]   5.00-6.00   sec   560 MBytes  4.70 Gbits/sec  405400 
[  5]   6.00-7.00   sec   658 MBytes  5.52 Gbits/sec  476440 
[  5]   7.00-8.00   sec   568 MBytes  4.76 Gbits/sec  410970 
[  5]   8.00-9.00   sec   645 MBytes  5.41 Gbits/sec  467250 
[  5]   9.00-10.00  sec   575 MBytes  4.82 Gbits/sec  416060 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  6.07 GBytes  5.21 Gbits/sec  0.000 ms  0/4498540 (0%)  sender
[  5]   0.00-10.00  sec  4.84 GBytes  4.16 Gbits/sec  0.001 ms  910802/4498534 (20%)  receiver
i have no clue why udp dosnt get the full speed but tcp works completely fine


now if i go through opnsense:
tcp:

Code: [Select]

Connecting to host 172.16.3.2, port 5201
[  5] local 172.16.4.21 port 33602 connected to 172.16.3.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   486 MBytes  4.08 Gbits/sec   61    184 KBytes       
[  5]   1.00-2.00   sec   496 MBytes  4.16 Gbits/sec  107    102 KBytes       
[  5]   2.00-3.00   sec   499 MBytes  4.18 Gbits/sec   64    225 KBytes       
[  5]   3.00-4.00   sec   489 MBytes  4.10 Gbits/sec    7    827 KBytes       
[  5]   4.00-5.00   sec   488 MBytes  4.09 Gbits/sec    2    824 KBytes       
[  5]   5.00-6.00   sec   488 MBytes  4.09 Gbits/sec    1   1.16 MBytes       
[  5]   6.00-7.00   sec   496 MBytes  4.16 Gbits/sec   15    611 KBytes       
[  5]   7.00-8.00   sec   496 MBytes  4.16 Gbits/sec   29    559 KBytes       
[  5]   8.00-9.00   sec   495 MBytes  4.15 Gbits/sec   49    300 KBytes       
[  5]   9.00-10.00  sec   498 MBytes  4.17 Gbits/sec   32    161 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.81 GBytes  4.14 Gbits/sec  367             sender
[  5]   0.00-10.00  sec  4.81 GBytes  4.13 Gbits/sec                  receiver


Code: [Select]

Connecting to host 172.16.3.2, port 5201
Reverse mode, remote host 172.16.3.2 is sending
[  5] local 172.16.4.21 port 45154 connected to 172.16.3.2 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   354 MBytes  2.97 Gbits/sec                 
[  5]   1.00-2.00   sec   312 MBytes  2.61 Gbits/sec                 
[  5]   2.00-3.00   sec   282 MBytes  2.37 Gbits/sec                 
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec                 
[  5]   4.00-5.00   sec   286 MBytes  2.40 Gbits/sec                 
[  5]   5.00-6.00   sec   290 MBytes  2.43 Gbits/sec                 
[  5]   6.00-7.00   sec   277 MBytes  2.33 Gbits/sec                 
[  5]   7.00-8.00   sec   305 MBytes  2.56 Gbits/sec                 
[  5]   8.00-9.00   sec   278 MBytes  2.33 Gbits/sec                 
[  5]   9.00-10.00  sec   306 MBytes  2.57 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.90 GBytes  2.49 Gbits/sec  9807             sender
[  5]   0.00-10.00  sec  2.90 GBytes  2.49 Gbits/sec                  receiver
udp:

Code: [Select]

Connecting to host 172.16.3.2, port 5201
[  5] local 172.16.4.21 port 51747 connected to 172.16.3.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   921 MBytes  7.73 Gbits/sec  667210 
[  5]   1.00-2.00   sec   647 MBytes  5.42 Gbits/sec  468260 
[  5]   2.00-3.00   sec   617 MBytes  5.17 Gbits/sec  446530 
[  5]   3.00-4.00   sec   615 MBytes  5.16 Gbits/sec  445700 
[  5]   4.00-5.00   sec   602 MBytes  5.05 Gbits/sec  435810 
[  5]   5.00-6.00   sec   611 MBytes  5.13 Gbits/sec  442640 
[  5]   6.00-7.00   sec   591 MBytes  4.96 Gbits/sec  428220 
[  5]   7.00-8.00   sec   618 MBytes  5.18 Gbits/sec  447510 
[  5]   8.00-9.00   sec   597 MBytes  5.01 Gbits/sec  432230 
[  5]   9.00-10.00  sec   623 MBytes  5.23 Gbits/sec  451190 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  6.29 GBytes  5.40 Gbits/sec  0.000 ms  0/4665300 (0%)  sender
[  5]   0.00-10.00  sec  4.67 GBytes  4.01 Gbits/sec  0.002 ms  1203096/4665237 (26%)  receiver



my initial thought is the cpu is to slow so i dissabled pf to see how it effected things the results are identical

[feld]

test net.isr.dispatch=deferred vs net.isr.dispatch=direct

[antsh415]

I have a similar system to yours:

Xeon E-2234
16GB RAM
Intel x550 adapter

I get around 7.5gbps for a single stream iperf3 run; any number of streams above that (-P 2 or higher) I easily max out the 10gbps limit of the adapter.

I just don't think *BSD distros will be able to get much higher performance than this, though this performance should be more than enough for the majority of use cases. If you really need >10gbps routing you should think about layer 3 switching or maybe try a linux distro.

I didn't have to change anything to reach the above speeds out of the box, though I did get a small boost implementing RSS as described on this page:

https://docs.opnsense.org/troubleshooting/performance.html

[antsh415]

I stand corrected, I don't know what changed, but with the latest update to 23.1.4 I am now able to max out my 10gig link using a single iperf3 stream.

This is achieved using inter-vlan routing using a single 10gig port as an uplink and one stream. It's a LXC container in a proxmox box routing over to a different vlan on a debian server.