2x25Gbps bnxt NIC (AOC-S25G-b2S) and ixl NIC (AOC-S25G-i2S) performance problem

[franco]

Sorry, I don't expect anyone will be looking at this.

Instead take a look here:

https://docs.opnsense.org/manual/interfaces_settings.html


Cheers,
Franco

[mimugmail]

Can you try 8 parallel streams? Quite sure one stream cant handle full bandwith with nic queue handling

[borys.ohnsorge]

Sorry, I don't expect anyone will be looking at this.

Instead take a look here:

https://docs.opnsense.org/manual/interfaces_settings.html

Cheers,
Franco

Can You advise (if You know of course) if the basic Business Support Package (No subscription) is sufficient to report and resolve such an issue or I have to solve this problem on my own? I have access to Broadcom cards and Intel cards and I think the solution to this problem will be good and valuable to me, to this product and to the rest of the OPNsense community.

Can you try 8 parallel streams?

Yes sure:
command: iperf3 -c 192.168.1.2 -P 8

Code Select Expand


[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.02 GBytes   880 Mbits/sec  125             sender
[  5]   0.00-10.00  sec  1.02 GBytes   879 Mbits/sec                  receiver
[  7]   0.00-10.00  sec  2.29 GBytes  1.97 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  2.29 GBytes  1.97 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  1.03 GBytes   885 Mbits/sec  123             sender
[  9]   0.00-10.00  sec  1.03 GBytes   884 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.28 GBytes  1.10 Gbits/sec   16             sender
[ 11]   0.00-10.00  sec  1.28 GBytes  1.10 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  2.95 GBytes  2.53 Gbits/sec    1             sender
[ 13]   0.00-10.00  sec  2.94 GBytes  2.53 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  2.28 GBytes  1.96 Gbits/sec    0             sender
[ 15]   0.00-10.00  sec  2.28 GBytes  1.96 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec   30             sender
[ 17]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1006 MBytes   844 Mbits/sec  106             sender
[ 19]   0.00-10.00  sec  1005 MBytes   843 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  13.2 GBytes  11.4 Gbits/sec  401             sender
[SUM]   0.00-10.00  sec  13.2 GBytes  11.3 Gbits/sec                  receiver

iperf Done.


screen of top command in attachement.

Quite sure one stream cant handle full bandwith with nic queue handling

Ubuntu 22.04 to Ubuntu 22.04 - can
Clear FreeBSD to Clear FreeBSD - also can
Please look at my earlier tests.

[FraLem]

By comparing both files, it looks to me as if they  are using different drivers for the ixl0 interface : <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2>  // ixl0: <Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k>

This generating different configuration of the network card.

Regards

[borys.ohnsorge]

By comparing both files, it looks to me as if they  are using different drivers for the ixl0 interface : <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2>  // ixl0: <Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k>

This generating different configuration of the network card.

Regards

I have tested on both drivers and it doesn't change anything.

[mimugmail]

When FreeBSD 14.1 to FreeBSD 14.1 is fester than OPN to OPN you should do a sysctl -a > sys.out and diff around. OPN will set defaults untouched since FreeBSD 11 (some of them). Just a guess ...

[borys.ohnsorge]

When FreeBSD 14.1 to FreeBSD 14.1 is fester than OPN to OPN you should do a sysctl -a > sys.out and diff around. OPN will set defaults untouched since FreeBSD 11 (some of them). Just a guess ...

This comparison is on 4 min film that I previously attached link to. I don't see any spectacular differences. But I don't know what I should be paying attention to.

[mimugmail]

Sorry, late to the party, didn't read every thread. Ok, I compared both.

1st test, set on OPN (only for testing):

net.inet.tcp.delayed_ack: 0 -> 1
net.inet.tcp.blackhole: 2 -> 0
net.inet.udp.blackhole: 1 -> 0
(net.inet.rss.enabled: 0 -> 1) enable in second test
net.isr.bindthreads: 1 -> 0
hw.ibrs_disable: 0 -> 1

If this doesn't help:

Firewall : Settings : Advanced -> Disable shared forwarding, reboot, test again

If this also doesn't help:

Screenshot of hw offloading

Also observing that F14 loads  newer driver where it has 4 queues compared to 8 in OPN. Just as an info.

[borys.ohnsorge]

Sorry, late to the party, didn't read every thread. Ok, I compared both.

1st test, set on OPN (only for testing):

net.inet.tcp.delayed_ack: 0 -> 1
net.inet.tcp.blackhole: 2 -> 0
net.inet.udp.blackhole: 1 -> 0
(net.inet.rss.enabled: 0 -> 1) enable in second test
net.isr.bindthreads: 1 -> 0
hw.ibrs_disable: 0 -> 1

If this doesn't help:

Firewall : Settings : Advanced -> Disable shared forwarding, reboot, test again

If this also doesn't help:

Screenshot of hw offloading

Also observing that F14 loads  newer driver where it has 4 queues compared to 8 in OPN. Just as an info.

OK, I will test it tomorrow morning.

[borys.ohnsorge]

Sorry, late to the party, didn't read every thread. Ok, I compared both.

1st test, set on OPN (only for testing):

net.inet.tcp.delayed_ack: 0 -> 1
net.inet.tcp.blackhole: 2 -> 0
net.inet.udp.blackhole: 1 -> 0
(net.inet.rss.enabled: 0 -> 1) enable in second test
net.isr.bindthreads: 1 -> 0
hw.ibrs_disable: 0 -> 1

If this doesn't help:

Firewall : Settings : Advanced -> Disable shared forwarding, reboot, test again

If this also doesn't help:

Screenshot of hw offloading

Also observing that F14 loads  newer driver where it has 4 queues compared to 8 in OPN. Just as an info.

After making the following changes it's a bit better, but it's still far from being on a pure FreeBSD.

Code Select Expand

root@admins-opn-1:~ # sysctl net.inet.tcp.delayed_ack
net.inet.tcp.delayed_ack: 1
root@admins-opn-1:~ # sysctl net.inet.tcp.blackhole
net.inet.tcp.blackhole: 0
root@admins-opn-1:~ # sysctl net.inet.udp.blackhole
net.inet.udp.blackhole: 0
root@admins-opn-1:~ # sysctl net.inet.rss.enabled
net.inet.rss.enabled: 0
root@admins-opn-1:~ # sysctl net.isr.bindthreads
net.isr.bindthreads: 0
root@admins-opn-1:~ # sysctl hw.ibrs_disable
hw.ibrs_disable: 1


iper3:

Code Select Expand

perf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 54166 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.02   sec   618 MBytes  5.10 Gbits/sec    0   1.32 MBytes       
[  5]   1.02-2.00   sec   659 MBytes  5.61 Gbits/sec    0   1.32 MBytes       
[  5]   2.00-3.00   sec   649 MBytes  5.45 Gbits/sec    0   1.32 MBytes       
[  5]   3.00-4.01   sec   654 MBytes  5.45 Gbits/sec    0   1.32 MBytes       
[  5]   4.01-5.06   sec   684 MBytes  5.44 Gbits/sec    0   1.32 MBytes       
[  5]   5.06-6.03   sec   628 MBytes  5.45 Gbits/sec   17   1021 KBytes       
[  5]   6.03-7.06   sec   672 MBytes  5.46 Gbits/sec    0   1.18 MBytes       
[  5]   7.06-8.03   sec   628 MBytes  5.43 Gbits/sec    0   1.18 MBytes       
[  5]   8.03-9.00   sec   640 MBytes  5.53 Gbits/sec    0   1.21 MBytes       
[  5]   9.00-10.02  sec   680 MBytes  5.58 Gbits/sec    0   1.28 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  6.36 GBytes  5.45 Gbits/sec   17             sender
[  5]   0.00-10.02  sec  6.36 GBytes  5.45 Gbits/sec                  receiver

iperf3 with -P 8:

Code Select Expand

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.62 GBytes  2.25 Gbits/sec   73             sender
[  5]   0.00-10.00  sec  2.62 GBytes  2.25 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  2.83 GBytes  2.43 Gbits/sec  116             sender
[  7]   0.00-10.00  sec  2.83 GBytes  2.43 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  4.06 GBytes  3.49 Gbits/sec   17             sender
[  9]   0.00-10.00  sec  4.06 GBytes  3.49 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec   503 MBytes   422 Mbits/sec  248             sender
[ 11]   0.00-10.00  sec   501 MBytes   420 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec  1.42 GBytes  1.22 Gbits/sec  302             sender
[ 13]   0.00-10.00  sec  1.42 GBytes  1.22 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  2.64 GBytes  2.27 Gbits/sec   46             sender
[ 15]   0.00-10.00  sec  2.64 GBytes  2.27 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  2.53 GBytes  2.18 Gbits/sec   10             sender
[ 17]   0.00-10.00  sec  2.53 GBytes  2.17 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  2.19 GBytes  1.88 Gbits/sec  357             sender
[ 19]   0.00-10.00  sec  2.19 GBytes  1.88 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  18.8 GBytes  16.1 Gbits/sec  1169             sender
[SUM]   0.00-10.00  sec  18.8 GBytes  16.1 Gbits/sec                  receiver

iperf Done.

speedtest:

Code Select Expand

root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 4.134 ms
Testing download speed................................................................................
Download: 7206.26 Mbit/s
Testing upload speed...
Upload: 6746.15 Mbit/s

I will add the test results after making changes that require a reboot in a moment.

After setting:

Code Select Expand

root@admins-opn-1:~ # sysctl net.inet.tcp.delayed_ack=1
net.inet.tcp.delayed_ack: 0 -> 1
root@admins-opn-1:~ # sysctl net.inet.tcp.blackhole=0
net.inet.tcp.blackhole: 2 -> 0
root@admins-opn-1:~ # sysctl net.inet.udp.blackhole=0
net.inet.udp.blackhole: 1 -> 0
root@admins-opn-1:~ # sysctl net.inet.rss.enabled
net.inet.rss.enabled: 1
root@admins-opn-1:~ # sysctl net.isr.bindthreads
net.isr.bindthreads: 0
root@admins-opn-1:~ # sysctl hw.ibrs_disable=1
hw.ibrs_disable: 0 -> 1


Code Select Expand

root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 5473 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   658 MBytes  5.50 Gbits/sec    0   1.16 MBytes       
[  5]   1.00-2.05   sec   717 MBytes  5.72 Gbits/sec    0   1.16 MBytes       
[  5]   2.05-3.05   sec   674 MBytes  5.67 Gbits/sec    0   1.16 MBytes       
[  5]   3.05-4.00   sec   638 MBytes  5.64 Gbits/sec    0   1.16 MBytes       
[  5]   4.00-5.05   sec   750 MBytes  5.99 Gbits/sec    0   1.16 MBytes       
[  5]   5.05-6.06   sec   689 MBytes  5.73 Gbits/sec    0   1.16 MBytes       
[  5]   6.06-7.00   sec   638 MBytes  5.70 Gbits/sec    0   1.16 MBytes       
[  5]   7.00-8.04   sec   704 MBytes  5.67 Gbits/sec    0   1.16 MBytes       
[  5]   8.04-9.00   sec   649 MBytes  5.68 Gbits/sec    0   1.16 MBytes       
[  5]   9.00-10.00  sec   687 MBytes  5.75 Gbits/sec    0   1.16 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.65 GBytes  5.71 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  6.65 GBytes  5.71 Gbits/sec                  receiver

iperf Done.

[mimugmail]

[SUM]   0.00-10.00  sec  18.8 GBytes  16.1 Gbits/sec  1169             sender
[SUM]   0.00-10.00  sec  18.8 GBytes  16.1 Gbits/sec                  receiver



We're getting closer ... slowly :)

[borys.ohnsorge]

[SUM]   0.00-10.00  sec  18.8 GBytes  16.1 Gbits/sec  1169             sender
[SUM]   0.00-10.00  sec  18.8 GBytes  16.1 Gbits/sec                  receiver
We're getting closer ... slowly :)

Very slowly, but I am still very grateful for your time, willingness and attempt to help.

OPNsense: (iperf3 -c x.x.x.x)


FreeBSD: (iperf3 -c x.x.x.x)


FreeBSD: (iperf3 -c x.x.x.x -P 8 )


FreeBSD: (iperf3 -c x.x.x.x -P 8 )

[mimugmail]

OPN -P8 also?
Did you try disable shared forwarding?

[borys.ohnsorge]

OPN -P8 also?

Yep

Did you try disable shared forwarding?

After disabling shared forwarding:

[borys.ohnsorge]

Interesting thing, below are the results from a freshly installed pfSense (no tuning/modifications):

Code Select Expand

[2.7.2-RELEASE][admin@pfSense1.home.local]/root: iperf3 -c x.x.x.2
Connecting to host x.x.x.2, port 5201
[  5] local x.x.x.1 port 63784 connected to x.x.x.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   640 MBytes  5.36 Gbits/sec   32    631 KBytes       
[  5]   1.00-2.00   sec   680 MBytes  5.71 Gbits/sec    0    653 KBytes       
[  5]   2.00-3.00   sec   678 MBytes  5.69 Gbits/sec    0    653 KBytes       
[  5]   3.00-4.00   sec   677 MBytes  5.67 Gbits/sec    1    559 KBytes       
[  5]   4.00-5.00   sec   676 MBytes  5.68 Gbits/sec    0    559 KBytes       
[  5]   5.00-6.00   sec   676 MBytes  5.68 Gbits/sec    0    559 KBytes       
[  5]   6.00-7.00   sec   674 MBytes  5.66 Gbits/sec    0    636 KBytes       
[  5]   7.00-8.00   sec   677 MBytes  5.67 Gbits/sec    1    547 KBytes       
[  5]   8.00-9.00   sec   682 MBytes  5.73 Gbits/sec    0    551 KBytes       
[  5]   9.00-10.00  sec   682 MBytes  5.72 Gbits/sec    1    425 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.58 GBytes  5.66 Gbits/sec   35             sender
[  5]   0.00-10.00  sec  6.58 GBytes  5.65 Gbits/sec                  receiver

iperf Done.


Code Select Expand

[2.7.2-RELEASE][admin@pfSense1.home.local]/root: pfctl -d
pf disabled
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: iperf3 -c x.x.x.2
Connecting to host x.x.x.2, port 5201
[  5] local x.x.x.1 port 22755 connected to x.x.x.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   797 MBytes  6.67 Gbits/sec    0    619 KBytes       
[  5]   1.00-2.00   sec   806 MBytes  6.76 Gbits/sec    0    619 KBytes       
[  5]   2.00-3.00   sec   796 MBytes  6.68 Gbits/sec    1    454 KBytes       
[  5]   3.00-4.00   sec   806 MBytes  6.76 Gbits/sec    0    454 KBytes       
[  5]   4.00-5.00   sec   809 MBytes  6.79 Gbits/sec    0    454 KBytes       
[  5]   5.00-6.00   sec   807 MBytes  6.77 Gbits/sec    0    454 KBytes       
[  5]   6.00-7.00   sec   805 MBytes  6.76 Gbits/sec    0    454 KBytes       
[  5]   7.00-8.00   sec   809 MBytes  6.78 Gbits/sec    0    454 KBytes       
[  5]   8.00-9.00   sec   802 MBytes  6.74 Gbits/sec    1    484 KBytes       
[  5]   9.00-10.00  sec   813 MBytes  6.81 Gbits/sec    0    533 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.86 GBytes  6.75 Gbits/sec    2             sender
[  5]   0.00-10.00  sec  7.86 GBytes  6.75 Gbits/sec                  receiver

iperf Done.


Code Select Expand

[2.7.2-RELEASE][admin@pfSense1.home.local]/root: pfctl -e
pf enabled
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: iperf3 -c x.x.x.2 -P 8
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec    0             sender
[  5]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec                  receiver
[  7]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec    0             sender
[  7]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec                  receiver
[  9]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec    0             sender
[  9]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec                  receiver
[ 11]   0.00-10.01  sec  1.09 GBytes   938 Mbits/sec    0             sender
[ 11]   0.00-10.01  sec  1.09 GBytes   938 Mbits/sec                  receiver
[ 13]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec    0             sender
[ 13]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec                  receiver
[ 15]   0.00-10.01  sec  1.09 GBytes   936 Mbits/sec    0             sender
[ 15]   0.00-10.01  sec  1.09 GBytes   936 Mbits/sec                  receiver
[ 17]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec    0             sender
[ 17]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec                  receiver
[ 19]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec    0             sender
[ 19]   0.00-10.01  sec  1.09 GBytes   937 Mbits/sec                  receiver
[SUM]   0.00-10.01  sec  8.73 GBytes  7.50 Gbits/sec    0             sender
[SUM]   0.00-10.01  sec  8.73 GBytes  7.50 Gbits/sec                  receiver

iperf Done.


Code Select Expand


[2.7.2-RELEASE][admin@pfSense1.home.local]/root: pfctl -d
pf disabled
[2.7.2-RELEASE][admin@pfSense1.home.local]/root: iperf3 -c x.x.x.2 -P 8
Connecting to host x.x.x.2, port 5201
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec   13             sender
[  5]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec   13             sender
[ 11]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec    0             sender
[ 13]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec    1             sender
[ 15]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec    0             sender
[ 17]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec    0             sender
[ 19]   0.00-10.00  sec  1.38 GBytes  1.19 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  11.1 GBytes  9.50 Gbits/sec   27             sender
[SUM]   0.00-10.00  sec  11.1 GBytes  9.50 Gbits/sec                  receiver

iperf Done.


What the ..., I know that it's still a long way to 25Gbps, which is what I am aiming for, but still, it's almost 2x better than on "pure" OPNsense.