2x25Gbps bnxt NIC (AOC-S25G-b2S) and ixl NIC (AOC-S25G-i2S) performance problem

[mimugmail]

There is a slight chance I get a test system with a bmxt 25G connected to a Mellanox Switch, will take a bit.

Problem is that iperf directly on OPN is terribly slow compared to traffic just passing through

[borys.ohnsorge]

Small success

Code Select Expand

root@admins-opn-1:~ # iperf3 -c x.x.x.2
Connecting to host x.x.x.2, port 5201
[ 5] local x.x.x.1 port 35935 connected to x.x.x.2 port 5201
[ ID] Interval      Transfer   Bitrate     Retr Cwnd
[ 5]  0.00-1.00  sec 1.43 GBytes 12.3 Gbits/sec  0  2.00 MBytes   
[ 5]  1.00-2.00  sec 1.44 GBytes 12.4 Gbits/sec  0  2.00 MBytes   
[ 5]  2.00-3.06  sec 1.54 GBytes 12.4 Gbits/sec  0  2.00 MBytes   
[ 5]  3.06-4.06  sec 1.43 GBytes 12.4 Gbits/sec  0  2.00 MBytes   
[ 5]  4.06-5.06  sec 1.43 GBytes 12.4 Gbits/sec  0  2.00 MBytes   
[ 5]  5.06-6.00  sec 1.36 GBytes 12.3 Gbits/sec  0  2.00 MBytes   
[ 5]  6.00-7.03  sec 1.47 GBytes 12.3 Gbits/sec  0  2.00 MBytes   
[ 5]  7.03-8.06  sec 1.48 GBytes 12.3 Gbits/sec  0  2.00 MBytes   
[ 5]  8.06-9.02  sec 1.37 GBytes 12.4 Gbits/sec  0  2.00 MBytes   
[ 5]  9.02-10.04 sec 1.47 GBytes 12.4 Gbits/sec  0  2.00 MBytes   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval      Transfer   Bitrate     Retr
[ 5]  0.00-10.04 sec 14.4 GBytes 12.3 Gbits/sec  0       sender
[ 5]  0.00-10.04 sec 14.4 GBytes 12.3 Gbits/sec         receiver

iperf Done.

Code Select Expand

root@admins-opn-1:~ # iperf3 -c x.x.x.2 -P8
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.86 GBytes  1.59 Gbits/sec  277             sender
[  5]   0.00-10.00  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  6.50 GBytes  5.58 Gbits/sec    2             sender
[  7]   0.00-10.00  sec  6.50 GBytes  5.58 Gbits/sec                  receiver
[  9]   0.00-10.00  sec   730 MBytes   612 Mbits/sec  112             sender
[  9]   0.00-10.00  sec   728 MBytes   611 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec  2.47 GBytes  2.12 Gbits/sec  247             sender
[ 11]   0.00-10.00  sec  2.47 GBytes  2.12 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  2.61 GBytes  2.24 Gbits/sec  267             sender
[ 13]   0.00-10.00  sec  2.61 GBytes  2.24 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  2.21 GBytes  1.90 Gbits/sec  270             sender
[ 15]   0.00-10.00  sec  2.21 GBytes  1.90 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  5.82 GBytes  5.00 Gbits/sec    0             sender
[ 17]   0.00-10.00  sec  5.82 GBytes  4.99 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  2.18 GBytes  1.87 Gbits/sec  270             sender
[ 19]   0.00-10.00  sec  2.18 GBytes  1.87 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  24.4 GBytes  20.9 Gbits/sec  1445             sender
[SUM]   0.00-10.00  sec  24.3 GBytes  20.9 Gbits/sec                  receiver

iperf Done.

[mimugmail]

And what did you change?

[borys.ohnsorge]

And what did you change?

I'm ashamed to admit it, but it seems that the main problem was the MTU settings. Currently, I achieved these results on a completely clean installation (couple of firewall rules).

On the switch side, I had the MTU set "fixed" to 9214 by my network team and I set the same on the OPNs side. Unfortunately, the effect of this was as you saw earlier. After setting the MTU to 8800 on the OPNsense side, everything started working except for pfSync (on unicast). This connection started working when I set the MTU to 1500 on the interface dedicated to HA and synchronization.

I still want to do some fine tuning and see if I can achieve even better results. When I have some free time, I will try to test the same settings for Broadcom cards as well.

[Patrick M. Hausen]

So called jumbo frames were a bad idea to begin with.

[borys.ohnsorge]

So called jumbo frames were a bad idea to begin with.

Yep.

[borys.ohnsorge]

More tuning&testing

Code Select Expand

hw.ibrs_disable=1
kern.ipc.maxsockbuf=614400000
net.isr.maxthreads=-1
net.isr.bindthreads=1
net.inet.rss.enabled=1
net.inet.tcp.soreceive_stream=1
net.inet.tcp.delayed_ack=1
net.inet.tcp.blackhole=0
net.inet.udp.blackhole=0


Code Select Expand

root@admins-opn-1:~ # iperf3 -c x.x.x.2
Connecting to host x.x.x.2, port 5201
[  5] local x.x.x.1 port 20279 connected to x.x.x.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.20 GBytes  18.9 Gbits/sec    0   2.01 MBytes       
[  5]   1.00-2.03   sec  2.30 GBytes  19.2 Gbits/sec    0   2.01 MBytes       
[  5]   2.03-3.03   sec  2.26 GBytes  19.3 Gbits/sec    0   2.01 MBytes       
[  5]   3.03-4.01   sec  2.22 GBytes  19.5 Gbits/sec    0   2.01 MBytes       
[  5]   4.01-5.00   sec  2.22 GBytes  19.2 Gbits/sec    0   2.01 MBytes       
[  5]   5.00-6.00   sec  2.22 GBytes  19.1 Gbits/sec    0   2.01 MBytes       
[  5]   6.00-7.00   sec  2.24 GBytes  19.2 Gbits/sec    0   2.01 MBytes       
[  5]   7.00-8.00   sec  2.24 GBytes  19.3 Gbits/sec    0   2.01 MBytes       
[  5]   8.00-9.01   sec  2.27 GBytes  19.4 Gbits/sec    0   2.01 MBytes       
[  5]   9.01-10.00  sec  2.25 GBytes  19.5 Gbits/sec    0   2.01 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  22.4 GBytes  19.3 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  22.4 GBytes  19.3 Gbits/sec                  receiver

iperf Done.


Code Select Expand

root@admins-opn-1:~ # iperf3 -c x.x.x.2 -P4
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  7.24 GBytes  6.21 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  6.68 GBytes  5.74 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  6.68 GBytes  5.73 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec    0             sender
[ 11]   0.00-10.00  sec  7.24 GBytes  6.21 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  28.4 GBytes  24.4 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  28.4 GBytes  24.4 Gbits/sec                  receiver


Code Select Expand

root@admins-opn-1:~ # iperf3 -c x.x.x.2 -P8
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.82 GBytes  1.56 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.81 GBytes  1.55 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  1.81 GBytes  1.55 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  3.61 GBytes  3.10 Gbits/sec    0             sender
[  9]   0.00-10.00  sec  3.61 GBytes  3.10 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  7.24 GBytes  6.22 Gbits/sec    0             sender
[ 11]   0.00-10.00  sec  7.24 GBytes  6.21 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  6.82 GBytes  5.85 Gbits/sec    0             sender
[ 13]   0.00-10.00  sec  6.82 GBytes  5.85 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.82 GBytes  1.56 Gbits/sec    0             sender
[ 15]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  3.63 GBytes  3.12 Gbits/sec    0             sender
[ 17]   0.00-10.00  sec  3.63 GBytes  3.12 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec    0             sender
[ 19]   0.00-10.00  sec  1.81 GBytes  1.56 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  28.6 GBytes  24.5 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  28.5 GBytes  24.5 Gbits/sec                  receiver

iperf Done.


[FraLem]

Looks good.

Would you share final interfaces configuration on Opnsense (ifconfig) as well as Driver used on the NIcs?

Thanks & regards

[borys.ohnsorge]

Looks good.

Would you share final interfaces configuration on Opnsense (ifconfig) as well as Driver used on the NIcs?

Thanks & regards

Yes Sure.

Code Select Expand

root@admins-opn-1:~ # ifconfig
ixl0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
    options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:11:b4
    media: Ethernet autoselect (25GBase-LR <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
ixl1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
    options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:11:b4
    hwaddr xx:xx:xx:xx:11:b5
    media: Ethernet autoselect (25GBase-CR <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:22:24
    media: Ethernet autoselect
    status: no carrier
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:22:25
    media: Ethernet autoselect
    status: no carrier
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
    options=0
    groups: enc
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pfsync0: flags=1000041<UP,RUNNING,LOWER_UP> metric 0 mtu 1500
    options=0
    syncdev: lagg0_vlan2 syncpeer: 192.168.2.249 maxupd: 128 defer: off version: 1400
    syncok: 1
    groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
    options=0
    groups: pflog
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
    options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
    ether xx:xx:xx:xx:11:b4
    hwaddr 00:00:00:00:00:00
    inet6 xx:xx:xx:xx%lagg0 prefixlen 64 scopeid 0x9
    laggproto lacp lagghash l2,l3
    laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lagg0_vlanXX: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: lagg0_vlanXX (opt2)
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet x.x.x.x netmask 0xfffffff0 broadcast x.x.x.x
    inet6 xx:xx:xx:xx%lagg0_vlanXX prefixlen 64 scopeid 0xa
    groups: vlan
    vlan: XX vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    description: lagg0_vlan2 (opt1)
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet 192.168.2.241 netmask 0xffffff00 broadcast 192.168.2.255
    inet6 xx:xx:xx:xx%lagg0_vlan2 prefixlen 64 scopeid 0xb
    groups: vlan
    vlan: 2 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0_vlan3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
    description: lagg0_vlan3 (opt6)
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet 10.255.5.2 netmask 0xffffff00 broadcast 10.255.5.255
    inet 10.255.5.1 netmask 0xffffff00 broadcast 10.255.5.255 vhid 91
    inet6 xx:xx:xx:xx%lagg0_vlan3 prefixlen 64 scopeid 0xc
    inet6 xx.xx.xx.xx::2 prefixlen 64
    inet6 xx.xx.xx.xx::1 prefixlen 64 vhid 92
    groups: vlan
    carp: MASTER vhid 91 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    carp: MASTER vhid 92 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    vlan: 3 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
lagg0_vlan4: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet6 xx:xx:xx:xx%lagg0_vlan4 prefixlen 64 scopeid 0xd
    groups: vlan
    vlan: 4 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lagg0_vlan52: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
    description: lagg0_vlan52 (wan)
    options=4000000<MEXTPG>
    ether xx:xx:xx:xx:11:b4
    inet x.x.x.201 netmask 0xffffffe0 broadcast x.x.x.223
    inet x.x.x.200 netmask 0xffffffe0 broadcast x.x.x.223 vhid 98
    inet6 xx:xx:xx:xx%lagg0_vlan52 prefixlen 64 scopeid 0xf
    inet6 xxxx:yyyy:zzzz::11 prefixlen 64
    inet6 xxxx:yyyy:zzzz::10 prefixlen 64 vhid 99
    groups: vlan
    carp: MASTER vhid 98 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    carp: MASTER vhid 99 advbase 1 advskew 0
          peer 224.0.0.18 peer6 ff02::12
    vlan: 52 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
root@admins-opn-1:~ #


Code Select Expand

FW: 9.30 default OPNsense ixl driver


Code Select Expand

root@admins-opn-1:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.ixl.1.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k