bnxt NIC (AOC-S25G-b2S - Broadcom BCM57414) stop working after upgrade

[borys.ohnsorge]

Hi,

I have just upgraded two freshly installed OPNsense 24.7.7 instances (connected to HA with CARP IPs, etc.) to 24.7.8, and after that, my network stopped working.

Network configuration:

WAN: 2x 25Gbps -> lagg0 -> lagg0_vlan52
LAN: 2x 25Gbps -> lagg0 -> lagg0_vlan5
pfsync: 2x 25Gbps -> lagg0 -> lagg0_vlan2

bnxt0 and bnxt1 are up/lower_up
status: active

lagg0 is up/lower_up but:
laggport: bnxt0 flags=0<>
laggport: bnxt1 flags=0<>
status: active

lagg0_vlan2 is up/lower_up
vlan: 2
vlanproto: 802.1q
vlanpcp: 0
parent interface: lagg0
groups: vlan
status: active

However, no network connection is working. I can't even ping the backup host on any interface.

When I reboot my server, I see many instances of the following error:
HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error

Did I do something wrong?

Regards
Borys O

[Patrick M. Hausen]

There is a problem with that line of NICs which is being discussed at least since January 2023:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133

You probably best get a different brand of cards.

EDIT: possibly a different firmware version might help:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245981

[borys.ohnsorge]

Is there any chance to "undo" upgrade?

[Patrick M. Hausen]

If you created a ZFS snapshot beforehand, yes. If you did not, you will have to reinstall.

[franco]

Unsure how this is directly related to 24.7.8 as bnxt driver wasn't touched. Testing the older kernel makes sense, but only for diagnosis:

# opnsene-update -kr 24.7.6


Cheers,
Franco

[borys.ohnsorge]

It's strange that the network works after shutdown for couple minutes. At the moment, one of the machines is working (the network is functional)."

Code: [Select]

root@admins-opn-1:~ # uname -a
FreeBSD admins-opn-1.redgelabs.com 14.1-RELEASE-p6 FreeBSD 14.1-RELEASE-p6 stable/24.7-n267939-fd5bc7f34e1 SMP amd64
root@admins-opn-1:~ # opnsense-version
OPNsense 24.7.8
root@admins-opn-1:~ # uptime
12:47PM  up 7 mins, 2 users, load averages: 0.11, 0.12, 0.07
root@admins-opn-1:~ #

Code: [Select]

root@admins-opn-1:~ # ifconfig lagg0
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 8800
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether 7c:c2:55:99:66:8e
hwaddr 00:00:00:00:00:00
inet6 fe80::7ec2:55ff:fe99:668e%lagg0 prefixlen 64 scopeid 0x9
laggproto lacp lagghash l2,l3
laggport: bnxt0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: bnxt1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Code: [Select]

root@admins-opn-1:~ # pciconf -l -BbcevV bnxt0
bnxt0@pci0:1:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x14e4 device=0x16d7 subvendor=0x15d9 subdevice=0x0928
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xdfe10000, size 65536, enabled
    bar   [18] = type Prefetchable Memory, range 64, base 0xdfd00000, size 1048576, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xdfe22000, size 8192, enabled
    cap 01[48] = powerspec 3  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 11[a0] = MSI-X supports 74 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x4a0]
    cap 10[ac] = PCI-Express 2 endpoint max data 256(512) FLR NS
                 max read 512
                 link x8(x8) speed 8.0(8.0) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[13c] = Serial 1 7cc255fffe99668e
    ecap 0004[150] = Power Budgeting 1
    ecap 0002[160] = VC 1 max VC0
    ecap 000b[180] = Vendor [1] ID 0000 Rev 0 Length 32
    ecap 0018[1b0] = LTR 1
    ecap 000e[1b8] = ARI 1
    ecap 0017[230] = TPH Requester 1
    ecap 0019[300] = PCIe Sec 1 lane errors 0
    ecap 001f[200] = Precision Time Measurement 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Supermicro Network Adapter'
    VPD ro PN  = 'AOC-S25G-b2S'
    VPD ro SN  = 'XXX'
    VPD ro V0  = '228.0.116.0'
    VPD ro V1  = '1.02'
    VPD ro V2  = '7CC25599668E'
    VPD ro V3  = '7CC25599668F'
    VPD ro V4  = '000000000000'
    VPD ro V5  = '000000000000'
    VPD ro VA  = '2'
root@admins-opn-1:~ # pciconf -l -BbcevV bnxt1
bnxt1@pci0:1:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x14e4 device=0x16d7 subvendor=0x15d9 subdevice=0x0928
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0xdfe00000, size 65536, enabled
    bar   [18] = type Prefetchable Memory, range 64, base 0xdfc00000, size 1048576, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0xdfe20000, size 8192, enabled
    cap 01[48] = powerspec 3  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 11[a0] = MSI-X supports 74 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x4a0]
    cap 10[ac] = PCI-Express 2 endpoint max data 256(512) FLR NS
                 max read 512
                 link x8(x8) speed 8.0(8.0) ClockPM disabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[13c] = Serial 1 7cc255fffe99668e
    ecap 0004[150] = Power Budgeting 1
    ecap 000b[180] = Vendor [1] ID 0000 Rev 0 Length 32
    ecap 000e[1b8] = ARI 1
    ecap 0017[230] = TPH Requester 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Supermicro Network Adapter'
    VPD ro PN  = 'AOC-S25G-b2S'
    VPD ro SN  = 'XXX'
    VPD ro V0  = '228.0.116.0'
    VPD ro V1  = '1.02'
    VPD ro V2  = '7CC25599668E'
    VPD ro V3  = '7CC25599668F'
    VPD ro V4  = '000000000000'
    VPD ro V5  = '000000000000'
    VPD ro VA  = '2'