[SOLVED] 2x25Gbps bnxt NIC and ixl NIC performance problem

Started by borys.ohnsorge, November 12, 2024, 09:38:48 AM

Previous topic - Next topic
Print
Go Down Pages1 2 3
User actions
November 12, 2024, 09:38:48 AM Last Edit: January 13, 2025, 12:21:22 PM by borys.ohnsorge
Hi,

I have just upgraded two freshly installed OPNsense 24.7.7 instances (connected to HA with CARP IPs, etc.) to 24.7.8, and after that, my network stopped working.

Network configuration:

WAN: 2x 25Gbps -> lagg0 -> lagg0_vlan52
LAN: 2x 25Gbps -> lagg0 -> lagg0_vlan5
pfsync: 2x 25Gbps -> lagg0 -> lagg0_vlan2

bnxt0 and bnxt1 are up/lower_up
status: active

lagg0 is up/lower_up but:
laggport: bnxt0 flags=0<>
laggport: bnxt1 flags=0<>
status: active

lagg0_vlan2 is up/lower_up
vlan: 2
vlanproto: 802.1q
vlanpcp: 0
parent interface: lagg0
groups: vlan
status: active

However, no network connection is working. I can't even ping the backup host on any interface.

When I reboot my server, I see many instances of the following error:
HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error

Did I do something wrong?

Regards
Borys O
November 12, 2024, 09:40:59 AM #1 Last Edit: November 12, 2024, 09:42:31 AM by Patrick M. Hausen
There is a problem with that line of NICs which is being discussed at least since January 2023:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133

You probably best get a different brand of cards.

EDIT: possibly a different firmware version might help:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245981
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)
November 12, 2024, 12:16:52 PM #2
Is there any chance to "undo" upgrade?
November 12, 2024, 12:18:34 PM #3
If you created a ZFS snapshot beforehand, yes. If you did not, you will have to reinstall.
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)
November 12, 2024, 12:34:20 PM #4
Unsure how this is directly related to 24.7.8 as bnxt driver wasn't touched. Testing the older kernel makes sense, but only for diagnosis:

# opnsene-update -kr 24.7.6


Cheers,
Franco
November 12, 2024, 12:48:05 PM #5 Last Edit: November 12, 2024, 01:03:13 PM by borys.ohnsorge
It's strange that the network works after shutdown for couple minutes. At the moment, one of the machines is working (the network is functional)."

Code Select
root@admins-opn-1:~ # uname -a
FreeBSD admins-opn-1.redgelabs.com 14.1-RELEASE-p6 FreeBSD 14.1-RELEASE-p6 stable/24.7-n267939-fd5bc7f34e1 SMP amd64
root@admins-opn-1:~ # opnsense-version
OPNsense 24.7.8
root@admins-opn-1:~ # uptime
12:47PM  up 7 mins, 2 users, load averages: 0.11, 0.12, 0.07
root@admins-opn-1:~ #

Code Select
root@admins-opn-1:~ # ifconfig lagg0
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 7c:c2:55:99:66:8e
hwaddr 00:00:00:00:00:00
inet6 fe80::7ec2:55ff:fe99:668e%lagg0 prefixlen 64 scopeid 0x9
laggproto lacp lagghash l2,l3
laggport: bnxt0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: bnxt1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Code Select
root@admins-opn-1:~ # pciconf -l -BbcevV bnxt0
bnxt0@pci0:1:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x14e4 device=0x16d7 subvendor=0x15d9 subdevice=0x0928
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xdfe10000, size 65536, enabled
    bar   [18] = type Prefetchable Memory, range 64, base 0xdfd00000, size 1048576, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xdfe22000, size 8192, enabled
    cap 01[48] = powerspec 3  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 11[a0] = MSI-X supports 74 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x4a0]
    cap 10[ac] = PCI-Express 2 endpoint max data 256(512) FLR NS
                 max read 512
                 link x8(x8) speed 8.0(8.0) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[13c] = Serial 1 7cc255fffe99668e
    ecap 0004[150] = Power Budgeting 1
    ecap 0002[160] = VC 1 max VC0
    ecap 000b[180] = Vendor [1] ID 0000 Rev 0 Length 32
    ecap 0018[1b0] = LTR 1
    ecap 000e[1b8] = ARI 1
    ecap 0017[230] = TPH Requester 1
    ecap 0019[300] = PCIe Sec 1 lane errors 0
    ecap 001f[200] = Precision Time Measurement 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Supermicro Network Adapter'
    VPD ro PN  = 'AOC-S25G-b2S'
    VPD ro SN  = 'XXX'
    VPD ro V0  = '228.0.116.0'
    VPD ro V1  = '1.02'
    VPD ro V2  = '7CC25599668E'
    VPD ro V3  = '7CC25599668F'
    VPD ro V4  = '000000000000'
    VPD ro V5  = '000000000000'
    VPD ro VA  = '2'
root@admins-opn-1:~ # pciconf -l -BbcevV bnxt1
bnxt1@pci0:1:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x14e4 device=0x16d7 subvendor=0x15d9 subdevice=0x0928
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xdfe00000, size 65536, enabled
    bar   [18] = type Prefetchable Memory, range 64, base 0xdfc00000, size 1048576, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xdfe20000, size 8192, enabled
    cap 01[48] = powerspec 3  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 11[a0] = MSI-X supports 74 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x4a0]
    cap 10[ac] = PCI-Express 2 endpoint max data 256(512) FLR NS
                 max read 512
                 link x8(x8) speed 8.0(8.0) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[13c] = Serial 1 7cc255fffe99668e
    ecap 0004[150] = Power Budgeting 1
    ecap 000b[180] = Vendor [1] ID 0000 Rev 0 Length 32
    ecap 000e[1b8] = ARI 1
    ecap 0017[230] = TPH Requester 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Supermicro Network Adapter'
    VPD ro PN  = 'AOC-S25G-b2S'
    VPD ro SN  = 'XXX'
    VPD ro V0  = '228.0.116.0'
    VPD ro V1  = '1.02'
    VPD ro V2  = '7CC25599668E'
    VPD ro V3  = '7CC25599668F'
    VPD ro V4  = '000000000000'
    VPD ro V5  = '000000000000'
    VPD ro VA  = '2'
December 03, 2024, 07:07:11 PM #6
Quote from: Patrick M. Hausen on November 12, 2024, 09:40:59 AM
There is a problem with that line of NICs which is being discussed at least since January 2023:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133

You probably best get a different brand of cards.

EDIT: possibly a different firmware version might help:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245981
Hi

After changing NIC to Intel XXV710 I still have network problems.

server1:
Code Select
Name
admins-opn-1
Versions
OPNsense 24.7.10_1-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15

root@admins-opn-1:~ # pciconf -l -BbcevV ixl0
ixl0@pci0:1:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x158b subvendor=0x15d9 subdevice=0x0978
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller XXV710 for 25GbE SFP28'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xdc000000, size 16777216, enabled
    bar   [1c] = type Prefetchable Memory, range 64, base 0xdd008000, size 32768, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 129 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
                 max read 512
                 link x8(x8) speed 8.0(8.0)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0110, VF RID Stride 0x0001
                     VF Device ID 0x154c
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
                     P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable
    ecap 0019[1d0] = PCIe Sec 1 lane errors 0xff
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Supermicro Network Adapter'
    VPD ro PN  = 'AOC-S25G-i2S                    '
    VPD ro V0  = '9.30 0x8000EA8A'
    VPD ro V1  = '2.00  '
    VPD ro SN  = '       '
    VPD ro VA  = '2'
    VPD ro V2  = ''
    VPD ro V3  = ''
    VPD ro V4  = '            '
    VPD ro V5  = '            '
    VPD rw VB  = ''
root@admins-opn-1:~ # pciconf -l -BbcevV ixl1
ixl1@pci0:1:0:1: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x158b subvendor=0x15d9 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller XXV710 for 25GbE SFP28'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xdb000000, size 16777216, enabled
    bar   [1c] = type Prefetchable Memory, range 64, base 0xdd000000, size 32768, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 129 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
                 max read 512
                 link x8(x8) speed 8.0(8.0)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x014f, VF RID Stride 0x0001
                     VF Device ID 0x154c
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
                     P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Supermicro Network Adapter'
    VPD ro PN  = 'AOC-S25G-i2S                    '
    VPD ro V0  = '9.30 0x8000EA8A'
    VPD ro V1  = '2.00  '
    VPD ro SN  = '       '
    VPD ro VA  = '2'
    VPD ro V2  = ''
    VPD ro V3  = ''
    VPD ro V4  = '            '
    VPD ro V5  = '            '
    VPD rw VB  = ''
root@admins-opn-1:~ #
root@admins-opn-1:~ #
root@admins-opn-1:~ # ifconfig lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether xx:xx:xx:xx:11:b4
hwaddr 00:00:00:00:00:00
laggproto lacp lagghash l2,l3
laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
root@admins-opn-1:~ # ifconfig lagg0_vlan52
lagg0_vlan52: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
description: WAN (wan)
options=4000000<MEXTPG>
ether xx:xx:xx:xx:11:b4
inet 192.168.1.1 netmask 0xffffffe0 broadcast 192.168.1.31
groups: vlan
vlan: 52 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
root@admins-opn-1:~ # kldstat
Id Refs Address                Size Name
1   61 0xffffffff80200000  1f63788 kernel
2    1 0xffffffff82164000    1e2c8 opensolaris.ko
3    1 0xffffffff82183000    165f0 if_lagg.ko
4    2 0xffffffff8219a000     3540 if_infiniband.ko
5    1 0xffffffff8219f000     ea58 if_bridge.ko
6    2 0xffffffff821ae000     8930 bridgestp.ko
7    1 0xffffffff821b7000     3c10 pflog.ko
8    3 0xffffffff821bb000    8d578 pf.ko
9    1 0xffffffff82249000    11a18 pfsync.ko
10    1 0xffffffff8225b000   5cd5e0 zfs.ko
11    1 0xffffffff82829000     aa30 if_gre.ko
12    1 0xffffffff82834000     4be0 if_enc.ko
13    1 0xffffffff82839000     fba8 carp.ko
14    1 0xffffffff83220000     2110 pchtherm.ko
15    1 0xffffffff83223000     3250 ichsmb.ko
16    1 0xffffffff83227000     2178 smbus.ko
17    1 0xffffffff8322a000     3360 uhid.ko
18    1 0xffffffff8322e000     4364 ums.ko
19    1 0xffffffff83233000     33c0 usbhid.ko
20    1 0xffffffff83237000     3380 hidbus.ko
21    1 0xffffffff8323b000     4850 nullfs.ko
root@admins-opn-1:~ #


server2:
Code Select
Name
admins-opn-2
Versions
OPNsense 24.7.10_1-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15

root@admins-opn-2:~ # pciconf -l -BbcevV ixl0
ixl0@pci0:1:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x158b subvendor=0x15d9 subdevice=0x0978
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller XXV710 for 25GbE SFP28'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xdc000000, size 16777216, enabled
    bar   [1c] = type Prefetchable Memory, range 64, base 0xdd008000, size 32768, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 129 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
                 max read 512
                 link x8(x8) speed 8.0(8.0)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0110, VF RID Stride 0x0001
                     VF Device ID 0x154c
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
                     P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable
    ecap 0019[1d0] = PCIe Sec 1 lane errors 0xff
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Supermicro Network Adapter'
    VPD ro PN  = 'AOC-S25G-i2S                    '
    VPD ro V0  = '9.30 0x8000EA8A'
    VPD ro V1  = '2.00  '
    VPD ro SN  = '       '
    VPD ro VA  = '2'
    VPD ro V2  = ''
    VPD ro V3  = ''
    VPD ro V4  = '            '
    VPD ro V5  = '            '
    VPD rw VB  = ''
root@admins-opn-2:~ # pciconf -l -BbcevV ixl1
ixl1@pci0:1:0:1: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x158b subvendor=0x15d9 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller XXV710 for 25GbE SFP28'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xdb000000, size 16777216, enabled
    bar   [1c] = type Prefetchable Memory, range 64, base 0xdd000000, size 32768, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 129 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
                 max read 512
                 link x8(x8) speed 8.0(8.0)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x014f, VF RID Stride 0x0001
                     VF Device ID 0x154c
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
                     P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Supermicro Network Adapter'
    VPD ro PN  = 'AOC-S25G-i2S                    '
    VPD ro V0  = '9.30 0x8000EA8A'
    VPD ro V1  = '2.00  '
    VPD ro SN  = '       '
    VPD ro VA  = '2'
    VPD ro V2  = ''
    VPD ro V3  = ''
    VPD ro V4  = '            '
    VPD ro V5  = '            '
    VPD rw VB  = ''
root@admins-opn-2:~ # ifconfig lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether yy:yy:yy:yy:17:ba
hwaddr 00:00:00:00:00:00
laggproto lacp lagghash l2,l3
laggport: ixl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: ixl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
root@admins-opn-2:~ # ifconfig lagg0_vlan52
lagg0_vlan52: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9214
description: WAN (wan)
options=4000000<MEXTPG>
ether yy:yy:yy:yy:17:ba
inet 192.168.1.2 netmask 0xffffffe0 broadcast 192.168.1.31
groups: vlan
vlan: 52 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
root@admins-opn-2:~ # kldstat
Id Refs Address                Size Name
1   61 0xffffffff80200000  1f63788 kernel
2    1 0xffffffff82164000     4be0 if_enc.ko
3    1 0xffffffff82169000    1e2c8 opensolaris.ko
4    1 0xffffffff82188000    165f0 if_lagg.ko
5    2 0xffffffff8219f000     3540 if_infiniband.ko
6    1 0xffffffff821a3000     ea58 if_bridge.ko
7    2 0xffffffff821b2000     8930 bridgestp.ko
8    1 0xffffffff821bb000   5cd5e0 zfs.ko
9    1 0xffffffff82789000    11a18 pfsync.ko
10    3 0xffffffff8279b000    8d578 pf.ko
11    1 0xffffffff82829000     fba8 carp.ko
12    1 0xffffffff82839000     3c10 pflog.ko
13    1 0xffffffff8283d000     aa30 if_gre.ko
14    1 0xffffffff83220000     2110 pchtherm.ko
15    1 0xffffffff83223000     3250 ichsmb.ko
16    1 0xffffffff83227000     2178 smbus.ko
17    1 0xffffffff8322a000     3360 uhid.ko
18    1 0xffffffff8322e000     4364 ums.ko
19    1 0xffffffff83233000     33c0 usbhid.ko
20    1 0xffffffff83237000     3380 hidbus.ko
21    1 0xffffffff8323b000     4850 nullfs.ko
root@admins-opn-2:~ #


iperf3 test result between this 2 server:
server1:
Code Select
root@admins-opn-1:~ # pfctl -d
pfctl: pf not enabled
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 2124 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.04   sec  0.00 Bytes  0.00 bits/sec    3   8.95 KBytes       
[  5]   1.04-2.05   sec  0.00 Bytes  0.00 bits/sec    2   8.95 KBytes       
[  5]   2.05-3.06   sec  0.00 Bytes  0.00 bits/sec    0   8.95 KBytes       
[  5]   3.06-4.06   sec  0.00 Bytes  0.00 bits/sec    1   8.95 KBytes       
[  5]   4.06-5.00   sec  0.00 Bytes  0.00 bits/sec    0   8.95 KBytes       
[  5]   5.00-6.06   sec  0.00 Bytes  0.00 bits/sec    1   8.95 KBytes       
[  5]   6.06-7.04   sec  0.00 Bytes  0.00 bits/sec    0   8.95 KBytes       
[  5]   7.04-8.04   sec  0.00 Bytes  0.00 bits/sec    0   8.95 KBytes       
[  5]   8.04-9.04   sec  0.00 Bytes  0.00 bits/sec    0   8.95 KBytes       
[  5]   9.04-10.04  sec  0.00 Bytes  0.00 bits/sec    1   8.95 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec    8             sender
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf Done.

server2:
Code Select
root@admins-opn-2:~ # pfctl -d
pfctl: pf not enabled
root@admins-opn-2:~ # iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.1.1, port 17875
[  5] local 192.168.1.2 port 5201 connected to 192.168.1.1 port 2124
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.06   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   1.06-2.06   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   2.06-3.02   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   3.02-4.03   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   4.03-5.05   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   5.05-6.03   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   6.03-7.05   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   7.05-8.04   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   8.04-9.06   sec  0.00 Bytes  0.00 bits/sec                 
[  5]   9.06-10.04  sec  0.00 Bytes  0.00 bits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------

On this hardware under Ubuntu 22.04 LTS (installed on one of disks) with same lag+vlan+ip addresses, iperf3 achieves about 23Gbps without any problems (look at attached screens).

Am I doing something wrong?

Tomorrow I will "disassemble" the lags and do a test on the vlan only ports. After that on full access port.

Any help/suggestion will be appreciated.

Regards
Borys
December 04, 2024, 10:03:57 PM #7
Small update:

Removed lag and vlan configuration. Currently WAN port is "access" port.

System up to date.
Code Select
root@admins-opn-1:~ # uname -a
FreeBSD admins-opn-1.redgelabs.com 14.1-RELEASE-p6 FreeBSD 14.1-RELEASE-p6 stable/24.7-n267979-0d692990122 SMP amd64

Code Select
root@admins-opn-1:~ # sysctl dev.ixl.0.fw_lldp=0
dev.ixl.0.fw_lldp: 1 -> 0
root@admins-opn-1:~ # sysctl dev.ixl.1.fw_lldp=0
dev.ixl.1.fw_lldp: 1 -> 0
root@admins-opn-1:~ # sysctl -a | grep bindthreads
net.isr.bindthreads: 1
root@admins-opn-1:~ # ifconfig ixl0
ixl0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether xx:xx:xx:xx:11:b4
inet 192.168.1.1 netmask 0xffffffe0 broadcast 192.168.1.31
media: Ethernet autoselect (25GBase-LR <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@admins-opn-1:~ #
root@admins-opn-1:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.1.%desc: Intel(R) I210 (Copper)
dev.igb.0.%desc: Intel(R) I210 (Copper)
dev.ixl.1.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k
root@admins-opn-1:~ #
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 61382 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.06   sec   313 MBytes  2.47 Gbits/sec  413    112 KBytes       
[  5]   1.06-2.00   sec   430 MBytes  3.83 Gbits/sec    0    723 KBytes       
[  5]   2.00-3.05   sec   477 MBytes  3.82 Gbits/sec    0   1.05 MBytes       
[  5]   3.05-4.06   sec   463 MBytes  3.85 Gbits/sec    0   1.05 MBytes       
[  5]   4.06-5.03   sec   441 MBytes  3.82 Gbits/sec    0   1.05 MBytes       
[  5]   5.03-6.00   sec   444 MBytes  3.83 Gbits/sec    0   1.05 MBytes       
[  5]   6.00-7.00   sec   457 MBytes  3.83 Gbits/sec    0   1.05 MBytes       
[  5]   7.00-8.02   sec   464 MBytes  3.81 Gbits/sec    0   1.05 MBytes       
[  5]   8.02-9.00   sec   448 MBytes  3.84 Gbits/sec    0   1.05 MBytes       
[  5]   9.00-10.00  sec   458 MBytes  3.84 Gbits/sec    0   1.05 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.29 GBytes  3.69 Gbits/sec  413             sender
[  5]   0.00-10.00  sec  4.29 GBytes  3.69 Gbits/sec                  receiver

iperf Done.


After adding Intels newest driver:
Code Select
root@admins-opn-1:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.1.%desc: Intel(R) I210 (Copper)
dev.igb.0.%desc: Intel(R) I210 (Copper)
dev.ixl.1.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 43611 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   432 MBytes  3.63 Gbits/sec   94    931 KBytes       
[  5]   1.00-2.00   sec   430 MBytes  3.61 Gbits/sec    0    931 KBytes       
[  5]   2.00-3.01   sec   433 MBytes  3.61 Gbits/sec    0    931 KBytes       
[  5]   3.01-4.00   sec   427 MBytes  3.60 Gbits/sec    0    931 KBytes       
[  5]   4.00-5.05   sec   450 MBytes  3.61 Gbits/sec    0    931 KBytes       
[  5]   5.05-6.05   sec   429 MBytes  3.60 Gbits/sec    0    931 KBytes       
[  5]   6.05-7.00   sec   411 MBytes  3.61 Gbits/sec    0    931 KBytes       
[  5]   7.00-8.00   sec   431 MBytes  3.61 Gbits/sec    0    931 KBytes       
[  5]   8.00-9.00   sec   431 MBytes  3.62 Gbits/sec    0    931 KBytes       
[  5]   9.00-10.05  sec   459 MBytes  3.67 Gbits/sec    0    931 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.05  sec  4.23 GBytes  3.62 Gbits/sec   94             sender
[  5]   0.00-10.05  sec  4.23 GBytes  3.61 Gbits/sec                  receiver

iperf Done.

Enable LLDP:
Code Select
root@admins-opn-1:~ # sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.1.%desc: Intel(R) I210 (Copper)
dev.igb.0.%desc: Intel(R) I210 (Copper)
dev.ixl.1.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2
root@admins-opn-1:~ # sysctl -a|grep lldp
dev.ixl.1.fw_lldp: 0
dev.ixl.0.fw_lldp: 0
root@admins-opn-1:~ # sysctl dev.ixl.0.fw_lldp=1
dev.ixl.0.fw_lldp: 0 -> 1
root@admins-opn-1:~ # sysctl dev.ixl.1.fw_lldp=1
dev.ixl.1.fw_lldp: 0 -> 1
root@admins-opn-1:~ # sysctl -a | grep lldp
dev.ixl.1.fw_lldp: 1
dev.ixl.0.fw_lldp: 1
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 30248 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   422 MBytes  3.53 Gbits/sec    0   1.08 MBytes       
[  5]   1.00-2.00   sec   411 MBytes  3.45 Gbits/sec    0   1.08 MBytes       
[  5]   2.00-3.00   sec   413 MBytes  3.46 Gbits/sec    0   1.08 MBytes       
[  5]   3.00-4.00   sec   414 MBytes  3.47 Gbits/sec    0   1.08 MBytes       
[  5]   4.00-5.00   sec   413 MBytes  3.46 Gbits/sec    0   1.08 MBytes       
[  5]   5.00-6.00   sec   414 MBytes  3.47 Gbits/sec    0   1.08 MBytes       
[  5]   6.00-7.01   sec   418 MBytes  3.47 Gbits/sec    0   1.08 MBytes       
[  5]   7.01-8.00   sec   410 MBytes  3.48 Gbits/sec    0   1.18 MBytes       
[  5]   8.00-9.00   sec   413 MBytes  3.47 Gbits/sec    0   1.18 MBytes       
[  5]   9.00-10.00  sec   413 MBytes  3.47 Gbits/sec    0   1.18 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  4.04 GBytes  3.47 Gbits/sec                  receiver

iperf Done.

Firmware and NVM version:
Code Select
ixl0: fw 9.130.73618 api 1.15 nvm 9.30 etid 8000ea8a oem 1.269.0

Speedtest-cli:
Code Select
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Exatel S.A (Warsaw): 9.006 ms
Download: 3924.67 Mbit/s
Upload: 1042.79 Mbit/s

Currently waiting for older fw from Supermicro to do more tests.

Regards
Borys
December 04, 2024, 10:49:20 PM #8 Last Edit: December 05, 2024, 11:35:29 AM by Patrick M. Hausen
Tough sh*t and I honestly have no idea at the moment. Your HW is quite top notch, so small chance to find anyone with more experience than you and I and that particular hardware. Sorry.
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)
December 05, 2024, 09:45:15 AM #9
I was managed to get Supermiro NIC fw 8.30, 8.50, 9.00 so I will do more tests.
December 05, 2024, 11:08:26 AM #10 Last Edit: December 05, 2024, 11:10:31 AM by borys.ohnsorge
FW: 9.00 default OPNsense ixl driver:
Code Select

dev.ixl.0.fw_lldp: 0
dev.ixl.0.fw_version: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d298 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k


Code Select
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 1.434 ms
Testing download speed................................................................................
Download: 4068.19 Mbit/s
Testing upload speed...
Upload: 1141.39 Mbit/s
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 54613 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.02   sec   468 MBytes  3.86 Gbits/sec   25    846 KBytes       
[  5]   1.02-2.06   sec   489 MBytes  3.93 Gbits/sec    0    846 KBytes       
[  5]   2.06-3.00   sec   439 MBytes  3.93 Gbits/sec    0    857 KBytes       
[  5]   3.00-4.04   sec   482 MBytes  3.91 Gbits/sec    0    857 KBytes       
[  5]   4.04-5.06   sec   478 MBytes  3.91 Gbits/sec    0    857 KBytes       
[  5]   5.06-6.01   sec   446 MBytes  3.92 Gbits/sec    0    857 KBytes       
[  5]   6.01-7.01   sec   468 MBytes  3.94 Gbits/sec    0    857 KBytes       
[  5]   7.01-8.02   sec   471 MBytes  3.92 Gbits/sec    0    857 KBytes       
[  5]   8.02-9.02   sec   469 MBytes  3.92 Gbits/sec    0    857 KBytes       
[  5]   9.02-10.00  sec   460 MBytes  3.92 Gbits/sec    0    857 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.56 GBytes  3.91 Gbits/sec   25             sender
[  5]   0.00-10.00  sec  4.56 GBytes  3.91 Gbits/sec                  receiver

iperf Done.


FW: 9.00 Intels 1.42.2 ixl driver:
Code Select

dev.ixl.0.fw_lldp: 0
dev.ixl.0.fw_version: fw 9.20.71847 api 1.15 nvm 9.00 etid 8000d298 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2


Code Select
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 1.274 ms
Testing download speed................................................................................
Download: 3682.38 Mbit/s
Testing upload speed...
Upload: 1488.75 Mbit/s
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 34854 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   414 MBytes  3.47 Gbits/sec   57   1.06 MBytes       
[  5]   1.00-2.00   sec   416 MBytes  3.49 Gbits/sec    0   1.10 MBytes       
[  5]   2.00-3.00   sec   422 MBytes  3.54 Gbits/sec    0   1.10 MBytes       
[  5]   3.00-4.03   sec   436 MBytes  3.54 Gbits/sec    0   1.19 MBytes       
[  5]   4.03-5.00   sec   409 MBytes  3.55 Gbits/sec    0   1.19 MBytes       
[  5]   5.00-6.04   sec   435 MBytes  3.52 Gbits/sec    0   1.19 MBytes       
[  5]   6.04-7.00   sec   407 MBytes  3.54 Gbits/sec    0   1.19 MBytes       
[  5]   7.00-8.00   sec   420 MBytes  3.52 Gbits/sec    0   1.19 MBytes       
[  5]   8.00-9.00   sec   425 MBytes  3.57 Gbits/sec    0   1.19 MBytes       
[  5]   9.00-10.00  sec   424 MBytes  3.55 Gbits/sec    0   1.19 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.11 GBytes  3.53 Gbits/sec   57             sender
[  5]   0.00-10.00  sec  4.11 GBytes  3.53 Gbits/sec                  receiver

iperf Done.



December 05, 2024, 12:00:45 PM #11
FW: 8.50 default OPNsense ixl driver:
Code Select

dev.ixl.0.fw_version: fw 8.5.67516 api 1.15 nvm 8.50 etid 8000b8b2 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k


Errors appeared in dmesg:
Code Select

ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT
ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT
ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT
ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT
ixl0: ixl_del_hw_filters: i40e_aq_remove_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_ENOENT


Code Select
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 1.273 ms
Download: 4036.40 Mbit/s
Upload: 1165.60 Mbit/s
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 30652 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   411 MBytes  3.44 Gbits/sec   13   1.04 MBytes       
[  5]   1.00-2.05   sec   432 MBytes  3.46 Gbits/sec    0   1.05 MBytes       
[  5]   2.05-3.02   sec   400 MBytes  3.45 Gbits/sec    0   1.11 MBytes       
[  5]   3.02-4.00   sec   406 MBytes  3.46 Gbits/sec    0   1.11 MBytes       
[  5]   4.00-5.06   sec   438 MBytes  3.46 Gbits/sec    0   1.11 MBytes       
[  5]   5.06-6.04   sec   401 MBytes  3.46 Gbits/sec    0   1.11 MBytes       
[  5]   6.04-7.05   sec   421 MBytes  3.47 Gbits/sec    0   1.19 MBytes       
[  5]   7.05-8.00   sec   393 MBytes  3.46 Gbits/sec    0   1.19 MBytes       
[  5]   8.00-9.00   sec   410 MBytes  3.45 Gbits/sec    0   1.19 MBytes       
[  5]   9.00-10.04  sec   429 MBytes  3.46 Gbits/sec    0   1.19 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  4.04 GBytes  3.46 Gbits/sec   13             sender
[  5]   0.00-10.04  sec  4.04 GBytes  3.46 Gbits/sec                  receiver

iperf Done.


FW: 8.50 Intels 1.42.2 ixl driver:
Code Select

dev.ixl.0.fw_version: fw 8.5.67516 api 1.15 nvm 8.50 etid 8000b8b2 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.14.2


No errors in dmesg:
Code Select
root@admins-opn-1:~ # speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (192.168.1.1)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 1.36 ms
Testing download speed................................................................................
Download: 3593.34 Mbit/s
Testing upload speed...
Upload: 1441.83 Mbit/s
root@admins-opn-1:~ # iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 12016 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.06   sec   405 MBytes  3.20 Gbits/sec   55    920 KBytes       
[  5]   1.06-2.00   sec   356 MBytes  3.17 Gbits/sec    0    980 KBytes       
[  5]   2.00-3.06   sec   400 MBytes  3.16 Gbits/sec    0    980 KBytes       
[  5]   3.06-4.04   sec   368 MBytes  3.16 Gbits/sec    0    989 KBytes       
[  5]   4.04-5.00   sec   361 MBytes  3.16 Gbits/sec    0   1020 KBytes       
[  5]   5.00-6.00   sec   377 MBytes  3.16 Gbits/sec    0   1020 KBytes       
[  5]   6.00-7.02   sec   385 MBytes  3.17 Gbits/sec    0   1020 KBytes       
[  5]   7.02-8.00   sec   368 MBytes  3.15 Gbits/sec    0   1020 KBytes       
[  5]   8.00-9.06   sec   400 MBytes  3.17 Gbits/sec    0   1020 KBytes       
[  5]   9.06-10.04  sec   370 MBytes  3.16 Gbits/sec    0   1020 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  3.70 GBytes  3.16 Gbits/sec   55             sender
[  5]   0.00-10.04  sec  3.70 GBytes  3.16 Gbits/sec                  receiver

iperf Done.


iperf3 to Ubuntu serwer with 10Gbps NIC:
Code Select

root@admins-opn-1:~ # iperf3 -c 192.168.1.3
Connecting to host 192.168.1.3, port 5201
[  5] local 192.168.1.1 port 44011 connected to 192.168.1.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   742 MBytes  6.22 Gbits/sec   45    268 KBytes       
[  5]   1.00-2.00   sec   722 MBytes  6.06 Gbits/sec   55    240 KBytes       
[  5]   2.00-3.00   sec   714 MBytes  5.98 Gbits/sec  110    157 KBytes       
[  5]   3.00-4.00   sec   714 MBytes  6.00 Gbits/sec   77   66.9 KBytes       
[  5]   4.00-5.02   sec   712 MBytes  5.86 Gbits/sec  101    339 KBytes       
[  5]   5.02-6.05   sec   743 MBytes  6.03 Gbits/sec   80    147 KBytes       
[  5]   6.05-7.04   sec   707 MBytes  6.05 Gbits/sec   66    377 KBytes       
[  5]   7.04-8.00   sec   704 MBytes  6.12 Gbits/sec   53    283 KBytes       
[  5]   8.00-9.00   sec   738 MBytes  6.17 Gbits/sec   55    318 KBytes       
[  5]   9.00-10.00  sec   726 MBytes  6.08 Gbits/sec   42    255 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.06 GBytes  6.06 Gbits/sec  684             sender
[  5]   0.00-10.01  sec  7.06 GBytes  6.06 Gbits/sec                  receiver

iperf Done.

December 05, 2024, 03:30:59 PM #12
@franco this has to be something with OPNsense just look:

Clean FreeBSD 14.1 install (on the same hardware - third disk :) ):
Code Select
# uname -a
FreeBSD admins-opn-1 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64

FW: 8.50 and default FreeBSD ixl driver:
Code Select

dev.ixl.0.fw_version: fw 8.5.67516 api 1.15 nvm 8.50 etid 8000b8b2 oem 1.268.0
dev.ixl.0.%desc: Intel(R) Ethernet Controller XXV710 for 25GbE SFP28 - 2.3.3-k


speedtest:
Code Select
# speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXX (x.x.x.x)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by NETIA S.A. (Warsaw) [0.32 km]: 2.695 ms
Testing download speed................................................................................
Download: 6703.99 Mbit/s
Testing upload speed...
Upload: 2069.06 Mbit/s


iperf3 to OPNsense up to date server:
Code Select
# iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 34871 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.06   sec   436 MBytes  3.44 Gbits/sec   67    883 KBytes       
[  5]   1.06-2.06   sec   408 MBytes  3.44 Gbits/sec    0    943 KBytes       
[  5]   2.06-3.06   sec   411 MBytes  3.45 Gbits/sec    0    943 KBytes       
[  5]   3.06-4.06   sec   411 MBytes  3.45 Gbits/sec    0    943 KBytes       
[  5]   4.06-5.01   sec   390 MBytes  3.45 Gbits/sec    0    943 KBytes       
[  5]   5.01-6.03   sec   420 MBytes  3.45 Gbits/sec    0    943 KBytes       
[  5]   6.03-7.06   sec   425 MBytes  3.45 Gbits/sec    0    943 KBytes       
[  5]   7.06-8.06   sec   411 MBytes  3.46 Gbits/sec    0    943 KBytes       
[  5]   8.06-9.06   sec   412 MBytes  3.45 Gbits/sec    0    943 KBytes       
[  5]   9.06-10.06  sec   410 MBytes  3.44 Gbits/sec    0   1019 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.06  sec  4.04 GBytes  3.45 Gbits/sec   67             sender
[  5]   0.00-10.06  sec  4.04 GBytes  3.45 Gbits/sec                  receiver

iperf Done.


iperf3 to same server as above but booted Ubuntu 22.04 (still same hardware)
Code Select
# iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 26181 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.06   sec   916 MBytes  7.22 Gbits/sec   58    341 KBytes       
[  5]   1.06-2.06   sec   686 MBytes  5.78 Gbits/sec   55   98.2 KBytes       
[  5]   2.06-3.06   sec   798 MBytes  6.69 Gbits/sec   62    486 KBytes       
[  5]   3.06-4.00   sec   806 MBytes  7.18 Gbits/sec   88    386 KBytes       
[  5]   4.00-5.06   sec  1.09 GBytes  8.83 Gbits/sec  102    285 KBytes       
[  5]   5.06-6.00   sec   991 MBytes  8.86 Gbits/sec   63    343 KBytes       
[  5]   6.00-7.06   sec  1.02 GBytes  8.26 Gbits/sec   20   9.91 KBytes       
[  5]   7.06-8.06   sec   977 MBytes  8.23 Gbits/sec   22    486 KBytes       
[  5]   8.06-9.06   sec   895 MBytes  7.51 Gbits/sec  165    193 KBytes       
[  5]   9.06-10.00  sec   922 MBytes  8.22 Gbits/sec   75    334 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  8.94 GBytes  7.68 Gbits/sec  710             sender
[  5]   0.00-10.08  sec  8.94 GBytes  7.62 Gbits/sec                  receiver

iperf Done.


Clean FreeBSD 14.1 iperf3 to Ubuntu with 10Gbps NIC:
Code Select
# iperf3 -c 192.168.1.3
Connecting to host 192.168.1.3, port 5201
[  5] local 192.168.1.1 port 24543 connected to 192.168.1.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.04   sec  1.04 GBytes  8.55 Gbits/sec   38    367 KBytes       
[  5]   1.04-2.04   sec  1.01 GBytes  8.64 Gbits/sec   26    691 KBytes       
[  5]   2.04-3.06   sec  1021 MBytes  8.42 Gbits/sec   60    694 KBytes       
[  5]   3.06-4.01   sec   963 MBytes  8.47 Gbits/sec   49    726 KBytes       
[  5]   4.01-5.00   sec   984 MBytes  8.36 Gbits/sec   54    284 KBytes       
[  5]   5.00-6.00   sec  1.05 GBytes  9.04 Gbits/sec   28    817 KBytes       
[  5]   6.00-7.06   sec  1.11 GBytes  9.01 Gbits/sec   26    394 KBytes       
[  5]   7.06-8.06   sec  1.01 GBytes  8.71 Gbits/sec   45    322 KBytes       
[  5]   8.06-9.06   sec  1015 MBytes  8.51 Gbits/sec   42    478 KBytes       
[  5]   9.06-10.06  sec  1.04 GBytes  8.93 Gbits/sec   23    123 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.06  sec  10.1 GBytes  8.67 Gbits/sec  391             sender
[  5]   0.00-10.14  sec  10.1 GBytes  8.60 Gbits/sec                  receiver

iperf Done.

iperf3 to same server as above but booted from fresh install of Clean FreeBSD 14.1 (still same hardware)
Code Select
# uname -a
FreeBSD admins-opn-2 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64


Code Select
# iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[  5] local 192.168.1.1 port 22759 connected to 192.168.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.72 GBytes  23.4 Gbits/sec    0   1.60 MBytes       
[  5]   1.00-2.00   sec  2.74 GBytes  23.5 Gbits/sec    0   1.60 MBytes       
[  5]   2.00-3.00   sec  2.74 GBytes  23.5 Gbits/sec    0   1.60 MBytes       
[  5]   3.00-4.00   sec  2.74 GBytes  23.5 Gbits/sec    0   1.60 MBytes       
[  5]   4.00-5.06   sec  2.91 GBytes  23.5 Gbits/sec    0   1.60 MBytes       
[  5]   5.06-6.06   sec  2.72 GBytes  23.5 Gbits/sec    0   1.60 MBytes       
[  5]   6.06-7.00   sec  2.59 GBytes  23.5 Gbits/sec    0   1.60 MBytes       
[  5]   7.00-8.00   sec  2.74 GBytes  23.5 Gbits/sec    0   1.61 MBytes       
[  5]   8.00-9.04   sec  2.86 GBytes  23.5 Gbits/sec    0   1.61 MBytes       
[  5]   9.04-10.06  sec  2.79 GBytes  23.5 Gbits/sec    0   1.61 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.06  sec  27.5 GBytes  23.5 Gbits/sec    0             sender
[  5]   0.00-10.06  sec  27.5 GBytes  23.5 Gbits/sec                  receiver

iperf Done.


So:

  • Ubuntu to Ubuntu - performance is ok about 23.5Gbps on 25Gbps NIC on both side.
  • OPNsense to OPNsense - performance is terrible about 3.5Gbps on 25Gbps NIC on both side.
  • OPNsense to Ubuntu - performance is about 7.5Gbps-8.5Gbps on 25Gbps NIC on both side.
  • OPNsense to Ubuntu(10Gbps) - performance is about 7.5Gbps-8.5Gbps on 25Gbps NIC on OPNsense side.
  • Clean FreeBSD 14.1 to OPNsense - performance is ok about 3.5Gbps on 25Gbps NIC on both side.
  • Clean FreeBSD 14.1 to Ubuntu - performance is about 7.5Gbps-8.5Gbps on 25Gbps NIC on both side.
  • Clean FreeBSD 14.1 to Ubuntu(10Gbps) - performance is about 7.5Gbps-8.5Gbps on 25Gbps NIC on OPNsense side.
  • Clean FreeBSD 14.1 to Clean FreeBSD 14.1 - performance is ok 23.5Gbps on 25Gbps NIC on both side. Same FW 8.5 different drivers (Intels 1.14.2 on first one and FreeBSD 2.3.3-k on second)

I can do more tests if You want.

Regards
Borys
December 05, 2024, 07:52:38 PM #13
Hardware acceleration settings maybe? They default to on in FreeBSD... In general difficult to tell, but likely a small setting somewhere.


Cheers,
Franco
December 05, 2024, 10:42:32 PM #14 Last Edit: December 05, 2024, 11:18:40 PM by borys.ohnsorge
Quote from: franco on December 05, 2024, 07:52:38 PM
Hardware acceleration settings maybe? They default to on in FreeBSD... In general difficult to tell, but likely a small setting somewhere.


Cheers,
Franco

I don't know what should I looking for, so I made diff between sysctl -a from Clear FreeBSD (sysctl_1) and OPNsense (sysctl_2) and I don't see any spectacular differences (files in attachments).

Here is the link to a video (4 minutes) from this diff:
https://owncloud.ohnsorge.pl/index.php/s/Navn1l86y397umv/download

Print
Go Up Pages1 2 3
User actions