Broadcom P425G not working (driver) on OPNsense 23.1

[Hans Peter Graf]

I recently installed a Broadcom P425G (4x 25Gbe) on a hardware running OPNsense 23.1. Unfortunately, there seems to be a problem with the drivers and no ethernet card is shown (ifconfig -a). 

I should mention that on Linux the P425G Ethernet card works, and that on Linux I was able to install the latest P425G firmware [1].

OPNsense (13.1-RELEASE-p7) has a driver/module: if_bnxt.ko . With this module loaded, no network device is found. Furthermore, I tried to compile the latest version of the if_bnxt.ko module from [1]. Even after unload the bundled driver and loading the new driver, no Ethernet device was found.

Do you have any suggestion ?


Here a portion of the output of pciconf -lbcevV :

Code: [Select]

none6@pci0:1:0:0: class=0x020000 rev=0x11 hdr=0x00 vendor=0x14e4 device=0x1801 subvendor=0x14e4 subdevice=0x4250
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'BCM57504 NetXtreme-E Ethernet Partition'
    class      = network
    subclass   = ethernet
    bar   [10] = type Prefetchable Memory, range 64, base 0x6045070000, size 65536, enabled
    bar   [18] = type Prefetchable Memory, range 64, base 0x6044000000, size 16777216, enabled
    bar   [20] = type Prefetchable Memory, range 64, base 0x60450b8000, size 32768, enabled
    cap 01[48] = powerspec 3  supports D0 D3  current D0
    cap 03[50] = VPD
    cap 11[a0] = MSI-X supports 74 messages
                 Table in map 0x20[0x0], PBA in map 0x20[0x4000]
    cap 10[ac] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 max read 4096
                 link x16(x16) speed 16.0(16.0) ClockPM enabled
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[13c] = Serial 1 84160cfffe648250
    ecap 0004[150] = Power Budgeting 1
    ecap 0002[160] = VC 1 max VC0
    ecap 000b[180] = Vendor [1] ID 0000 Rev 0 Length 32
    ecap 0018[1b0] = LTR 1
    ecap 000e[1b8] = ARI 1
    ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 8 supported
                     First VF RID Offset 0x0100, VF RID Stride 0x0001
                     VF Device ID 0x1806
                     Page Sizes: 4096 (enabled), 8192, 16384, 32768, 65536, 131072, 262144, 524288, 1048576, 4194304
    ecap 0017[230] = TPH Requester 1
    ecap 0019[300] = PCIe Sec 1 lane errors 0
    ecap 001f[200] = Precision Time Measurement 1
    ecap 0026[358] = Physical Layer 16.0 GT/s 1
    ecap 0027[388] = Lane Margining at Receiver 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error
    VPD ident  = 'Broadcom NetXtreme E-Series Quad-port 25Gb SFP28 PCIe Ethernet Adapter'
    VPD ro PN  = 'BCM957504-P425G'
    VPD ro MN  = '14E4'
    VPD ro V0  = '226.1.107.1'
    VPD ro V1  = '226.0.135.0'
    VPD ro V3  = '226.0.145.1'
    VPD ro V6  = '226.0.135.0'
    VPD ro V7  = '0.0.0'
    VPD ro V9  = '0.0.0'
    VPD ro VB  = 'REV016DEV869DEV917'
    VPD ro SN  = 'P425G2215003XXXX'




[1] https://www.broadcom.com/products/ethernet-connectivity/network-adapters/p425g

[Thomas Niedermeier]

Hello,

yes these Broadcom Cards don't work with OPNsense 23.1, we discovered this issue internally some time ago.
The FreeBSD driver for these quite new cards is too old to support them, but Broadcom updated it and with FreeBSD 13.2-Release these cards are recognised and work quite good.
I only tested it with plain FreeBSD 13.2-Release, because OPNsense is based on 13.1-Release.
But Franco tweeted in May that he successfully booted OPNsense with 13.2, so I hope future OPNsense 23.7 is based on 13.2, that would fix the issue with these cards.

Best regards,
Thomas

Output from FreeBSD 13.2-Release:

root@:~ # sysctl -a |grep -i broadcom
 
bnxt2: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc030000-0x303dbc03ffff,0x303dbb000000-0x303dbbffffff,0x303dbc058000-0x303dbc05ffff irq 88 at device 0.0 numa-domain 0 on pci11

--> BCM57504 is the IC of the P425G

New driver version in FreeBSD 13.2-Release:
root@:~ # sysctl -a | grep -E 'dev.*.iflib.driver_version' -A 5
dev.bnxt.7.iflib.driver_version: 2.20.0.1
dev.bnxt.7.%domain: 0
dev.bnxt.7.%parent: pci12
dev.bnxt.7.%pnpinfo: vendor=0x14e4 device=0x1750 subvendor=0x14e4 subdevice=0x2100 class=0x020000
dev.bnxt.7.%location: slot=0 function=1 dbsf=pci0:2:0:1
dev.bnxt.7.%driver: bnxt

[franco]

The latest 13.2/23.7 OS sets are:

# opnsense-update -zbkr 23.7.d_24
# opnsense-shell reboot

It would be nice to get a confirmation if these have been fixed indeed.


Cheers,
Franco

[Thomas Niedermeier]

Hi Franco,

thanks for the new sets!
I will test it tomorrow with the Broadcom cards and give feedback.

Best regards,
Thomas

[Thomas Niedermeier]

We tested the new kernel, works fine!
Driver is updated to 2.20.0.1. We did some quick tests, connected bnxt2 (first NIC on the P425G) to the network and it has received an address via dhcp.

root@OPNsense:~ # sysctl -a | grep -E 'dev.*.iflib.driver_version' -A 5
dev.bnxt.2.iflib.driver_version: 2.20.0.1
dev.bnxt.2.%domain: 0
dev.bnxt.2.%parent: pci10
dev.bnxt.2.%pnpinfo: vendor=0x14e4 device=0x1751 subvendor=0x14e4 subdevice=0x4250 class=0x020000
dev.bnxt.2.%location: slot=0 function=0 dbsf=pci0:1:0:0
dev.bnxt.2.%driver: bnxt

Quad Port P425G 25G Card (# sysctl -a |grep -i broadcom):

bnxt2: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc030000-0x303dbc03ffff,0x303dbb000000-0x303dbbffffff,0x303dbc058000-0x303dbc05ffff irq 88 at device 0.0 numa-domain 0 on pci10
bnxt3: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc020000-0x303dbc02ffff,0x303dba000000-0x303dbaffffff,0x303dbc050000-0x303dbc057fff irq 89 at device 0.1 numa-domain 0 on pci10
bnxt4: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc010000-0x303dbc01ffff,0x303db9000000-0x303db9ffffff,0x303dbc048000-0x303dbc04ffff irq 88 at device 0.2 numa-domain 0 on pci10
bnxt5: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc000000-0x303dbc00ffff,0x303db8000000-0x303db8ffffff,0x303dbc040000-0x303dbc047fff irq 89 at device 0.3 numa-domain 0 on pci10

Dual Port 100G P2100G Card:

bnxt6: <Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbf010000-0x303dbf01ffff,0x303dbe000000-0x303dbeffffff,0x303dbf028000-0x303dbf02ffff irq 90 at device 0.0 numa-domain 0 on pci11
bnxt7: <Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbf000000-0x303dbf00ffff,0x303dbd000000-0x303dbdffffff,0x303dbf020000-0x303dbf027fff irq 91 at device 0.1 numa-domain 0 on pci11

[franco]

ok great