Show Posts
1
Hardware and Performance / opnsense 10gb low performance higher cpu on iperf and nfs server
« on: March 14, 2023, 09:07:23 pm »
for the better part of a year i have been trying to get opnsense to do 10gb
i have a dell r330 with a Intel(R) Xeon(R) CPU E3-1270 v5 @ 3.60GHz and 8 gb of ram and a mellenox x3 card as the bare metal opnsense box
this is a big issue as this is holding back alot of my devices that need the full throughput
id also like to add when doing iperf tests when on same vlan the cpu useage on one core is 60% ish on the iperf server
when going thru opnsense the cpu is at 100%
why would going through opnsense cause higher cpu usage while delivering slower performance?
edit:
after more testing when going thru opnsense cause the nfs server to peg a core at 100% and peek tranfer at 3-4gbps wtf is going on
i have two machines on two diffrent vlans.
when on the same vlan i get via iperf results:
tcp:
now if i go through opnsense:
tcp:
my initial thought is the cpu is to slow so i dissabled pf to see how it effected things the results are identical
i have a dell r330 with a Intel(R) Xeon(R) CPU E3-1270 v5 @ 3.60GHz and 8 gb of ram and a mellenox x3 card as the bare metal opnsense box
this is a big issue as this is holding back alot of my devices that need the full throughput
id also like to add when doing iperf tests when on same vlan the cpu useage on one core is 60% ish on the iperf server
when going thru opnsense the cpu is at 100%
why would going through opnsense cause higher cpu usage while delivering slower performance?
edit:
after more testing when going thru opnsense cause the nfs server to peg a core at 100% and peek tranfer at 3-4gbps wtf is going on
i have two machines on two diffrent vlans.
when on the same vlan i get via iperf results:
tcp:
Code: [Select]
Connecting to host 172.16.3.2, port 5201
[ 5] local 172.16.3.4 port 53690 connected to 172.16.3.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.09 GBytes 9.39 Gbits/sec 242 1.02 MBytes
[ 5] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec 8 1004 KBytes
[ 5] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec 8 932 KBytes
[ 5] 3.00-4.00 sec 1.09 GBytes 9.37 Gbits/sec 8 980 KBytes
[ 5] 4.00-5.00 sec 1.09 GBytes 9.38 Gbits/sec 8 986 KBytes
[ 5] 5.00-6.00 sec 1.09 GBytes 9.37 Gbits/sec 8 980 KBytes
[ 5] 6.00-7.00 sec 1.09 GBytes 9.37 Gbits/sec 8 986 KBytes
[ 5] 7.00-8.00 sec 1.09 GBytes 9.38 Gbits/sec 8 928 KBytes
[ 5] 8.00-9.00 sec 1.09 GBytes 9.37 Gbits/sec 8 1.03 MBytes
[ 5] 9.00-10.00 sec 1.09 GBytes 9.38 Gbits/sec 8 923 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec 314 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec receiver
Code: [Select]
Connecting to host 172.16.3.2, port 5201
Reverse mode, remote host 172.16.3.2 is sending
[ 5] local 172.16.3.4 port 55870 connected to 172.16.3.2 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.09 GBytes 9.36 Gbits/sec
[ 5] 1.00-2.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 5] 2.00-3.00 sec 1.09 GBytes 9.38 Gbits/sec
[ 5] 3.00-4.00 sec 1.09 GBytes 9.36 Gbits/sec
[ 5] 4.00-5.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 5] 5.00-6.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 5] 6.00-7.00 sec 1.09 GBytes 9.35 Gbits/sec
[ 5] 7.00-8.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 5] 8.00-9.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 5] 9.00-10.00 sec 1.09 GBytes 9.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.37 Gbits/sec 128 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.37 Gbits/sec receiver
udp:Code: [Select]
Connecting to host 172.16.3.2, port 5201
[ 5] local 172.16.3.4 port 58135 connected to 172.16.3.2 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 778 MBytes 6.53 Gbits/sec 563560
[ 5] 1.00-2.00 sec 563 MBytes 4.72 Gbits/sec 407500
[ 5] 2.00-3.00 sec 648 MBytes 5.44 Gbits/sec 469190
[ 5] 3.00-4.00 sec 568 MBytes 4.77 Gbits/sec 411550
[ 5] 4.00-5.00 sec 650 MBytes 5.45 Gbits/sec 470620
[ 5] 5.00-6.00 sec 560 MBytes 4.70 Gbits/sec 405400
[ 5] 6.00-7.00 sec 658 MBytes 5.52 Gbits/sec 476440
[ 5] 7.00-8.00 sec 568 MBytes 4.76 Gbits/sec 410970
[ 5] 8.00-9.00 sec 645 MBytes 5.41 Gbits/sec 467250
[ 5] 9.00-10.00 sec 575 MBytes 4.82 Gbits/sec 416060
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 6.07 GBytes 5.21 Gbits/sec 0.000 ms 0/4498540 (0%) sender
[ 5] 0.00-10.00 sec 4.84 GBytes 4.16 Gbits/sec 0.001 ms 910802/4498534 (20%) receiver
i have no clue why udp dosnt get the full speed but tcp works completely finenow if i go through opnsense:
tcp:
Code: [Select]
Connecting to host 172.16.3.2, port 5201
[ 5] local 172.16.4.21 port 33602 connected to 172.16.3.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 486 MBytes 4.08 Gbits/sec 61 184 KBytes
[ 5] 1.00-2.00 sec 496 MBytes 4.16 Gbits/sec 107 102 KBytes
[ 5] 2.00-3.00 sec 499 MBytes 4.18 Gbits/sec 64 225 KBytes
[ 5] 3.00-4.00 sec 489 MBytes 4.10 Gbits/sec 7 827 KBytes
[ 5] 4.00-5.00 sec 488 MBytes 4.09 Gbits/sec 2 824 KBytes
[ 5] 5.00-6.00 sec 488 MBytes 4.09 Gbits/sec 1 1.16 MBytes
[ 5] 6.00-7.00 sec 496 MBytes 4.16 Gbits/sec 15 611 KBytes
[ 5] 7.00-8.00 sec 496 MBytes 4.16 Gbits/sec 29 559 KBytes
[ 5] 8.00-9.00 sec 495 MBytes 4.15 Gbits/sec 49 300 KBytes
[ 5] 9.00-10.00 sec 498 MBytes 4.17 Gbits/sec 32 161 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.81 GBytes 4.14 Gbits/sec 367 sender
[ 5] 0.00-10.00 sec 4.81 GBytes 4.13 Gbits/sec receiver
Code: [Select]
Connecting to host 172.16.3.2, port 5201
Reverse mode, remote host 172.16.3.2 is sending
[ 5] local 172.16.4.21 port 45154 connected to 172.16.3.2 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 354 MBytes 2.97 Gbits/sec
[ 5] 1.00-2.00 sec 312 MBytes 2.61 Gbits/sec
[ 5] 2.00-3.00 sec 282 MBytes 2.37 Gbits/sec
[ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 4.00-5.00 sec 286 MBytes 2.40 Gbits/sec
[ 5] 5.00-6.00 sec 290 MBytes 2.43 Gbits/sec
[ 5] 6.00-7.00 sec 277 MBytes 2.33 Gbits/sec
[ 5] 7.00-8.00 sec 305 MBytes 2.56 Gbits/sec
[ 5] 8.00-9.00 sec 278 MBytes 2.33 Gbits/sec
[ 5] 9.00-10.00 sec 306 MBytes 2.57 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.90 GBytes 2.49 Gbits/sec 9807 sender
[ 5] 0.00-10.00 sec 2.90 GBytes 2.49 Gbits/sec receiver
udp:Code: [Select]
Connecting to host 172.16.3.2, port 5201
[ 5] local 172.16.4.21 port 51747 connected to 172.16.3.2 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 921 MBytes 7.73 Gbits/sec 667210
[ 5] 1.00-2.00 sec 647 MBytes 5.42 Gbits/sec 468260
[ 5] 2.00-3.00 sec 617 MBytes 5.17 Gbits/sec 446530
[ 5] 3.00-4.00 sec 615 MBytes 5.16 Gbits/sec 445700
[ 5] 4.00-5.00 sec 602 MBytes 5.05 Gbits/sec 435810
[ 5] 5.00-6.00 sec 611 MBytes 5.13 Gbits/sec 442640
[ 5] 6.00-7.00 sec 591 MBytes 4.96 Gbits/sec 428220
[ 5] 7.00-8.00 sec 618 MBytes 5.18 Gbits/sec 447510
[ 5] 8.00-9.00 sec 597 MBytes 5.01 Gbits/sec 432230
[ 5] 9.00-10.00 sec 623 MBytes 5.23 Gbits/sec 451190
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 6.29 GBytes 5.40 Gbits/sec 0.000 ms 0/4665300 (0%) sender
[ 5] 0.00-10.00 sec 4.67 GBytes 4.01 Gbits/sec 0.002 ms 1203096/4665237 (26%) receiver
my initial thought is the cpu is to slow so i dissabled pf to see how it effected things the results are identical