[SOLVED] BCM57416 NetXtreme-E 10G RDMA Ethernet Controller not recognized

[flowerpower]

Hi to all,
I have a DELL PowerEdge R340 with 4 network cards:
2 x Embedded Gigabit Network cards
2x BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller
OPNsense 21.7 installed (hardenedbsd 12.1)

Problem: OPNsense not recognize the BCM57416 10G networks cards
The OS found the hardware interfaces like in the image attached but seems that not has the driver for these cards.

Instead pfSense 2.5.2 (freebsd 12.2) found correctly these cards.

Why OPNSense doesn't recognize these cards? I can solve this problem?

Thanks in advance to those who will help me

[flowerpower]

"ifconfig -a" result in the attached image

[opnfwb]

I don't have these specific cards so I can't give any direct experience with them. I did some searching around though.

It looks like the pfSense DEVs added the bnxt driver starting with 2.5.

There's another thread here where an OPNsense user claimed they resolved it with this:

Code Select Expand

kldload if_bnxt

Link to that thread: https://forum.opnsense.org/index.php?topic=21447.0

[flowerpower]

Hi, thanks for the answer.
I noticed that if I unload and reload the driver the OS recognize the network cards:

Code Select Expand


kldunload if_bnxt.ko
kldload if_bnxt.ko


But if I reboot the server the problem happens again...
I have attached an image with red circled the 10g network cards recognized by the OS after the unload and reload the driver

What can I do for resolve?

[Patrick M. Hausen]

Add a tunable named if_bnxt_load with value YES.

[flowerpower]

Hi,
I have created a file /boot/loader.conf.local with this row inside:

Code Select Expand

if_bnxt_load="YES"

than I have rebooted the server and in the attached image you can see what I see with the command

Code Select Expand

kldstat -v

but the OS doesn't recognize the network cards. I can use the 10G networks cards only if I unload and reload the driver... really strange

[Patrick M. Hausen]

Hi,
I have created a file /boot/loader.conf.local with this row inside:

Code Select Expand

if_bnxt_load="YES"

Why don't you use the UI for that? I'd recommend to stick to documented and supported methods of configuration ;)

Wrt to your problem: the driver has a history of minor bugs and the base FreeBSD of OPNsense is a bit outdated (12.1) - I would try to work around for now and check back when 22.1 is out which supposedly comes with an up to date base OS.

[flowerpower]

Finally I have solved the problem!!

I have put inside /boot/loader.conf.local  (if not exists you need to create this file) the following lines:

Code Select Expand

if_bnxt_load="YES"
dev.bnxt.0.iflib.override_nrxqs="8"
dev.bnxt.0.iflib.override_ntxqs="8"
dev.bnxt.1.iflib.override_nrxqs="8"
dev.bnxt.1.iflib.override_ntxqs="8"

Like reported on this FreeBSD bugzilla thread:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245981
It's a bug with FreeBSD 12.1 (hardenedBSD of opnsense 21.7 is based on this FreeBSD version) and the Broadcom NetXtreme-E DELL firmware (installed on my PowerEdge R340)

The number of queue size is derived from the number of cores on socket connected to the controller, like described here: https://nxmnpg.lemoda.net/4/bnxt


I hope this  can help someone

[Patrick M. Hausen]

Finally I have solved the problem!!

Great! And thanks for sharing.

I have put inside /boot/loader.conf.local  (if not exists you need to create this file) the following lines:

If you use the UI (System > Settings > Tunables) for that, your changes will be included in configuration backups and are guaranteed to persist across updates. If you edit files on the system directly, anything can happen ...

[flowerpower]

I have edited loader.conf.local not overwritten by the system. But yes I will use the opnsense UI. Thanks!

[Galicia86]

OPNsense not recognize the BCM57416 10G networks cards. I would like to know exactly how to make OPNSense activate the interfaces? Someone would have an OPNSEnse iso with the activation of interfaces. I'm a novice and I don't know how to create the Loader.conf.local file

[Patrick M. Hausen]

Yet another thread you are taking this problem to ...

I answered you via email. You need to login to your OPNsense with SSH. Command line. If you cannot do that we will not get the necessary debug output to have the FreeBSD developers look at why your particular card is not recognised. Get some help from someone at your location who knows how to use the command line.

Don't spam hundreds of mailing list participants with utter gibberish and don't discuss the same problem in more than one forum thread.

Please ...

Please report back when you have command line access to your system.
One question for now: according to that other thread you did add the tunable if_bnxt_load with value YES via the UI? Is that the case?

We need the output of

kldstat

to verify and the output of

ifconfig -a

to see if the interface is really not recognised with the kernel module loaded. Only if both questions are answered positive - kernel module is loaded and interface is not recognised - can we take all of this information including the output of

pciconf -lv

to the developers.

Hope that clears up what is expected from you and how we can proceed to help you. Nobody knows what particularly is wrong with your system unless you provide this information in exactly the way that I explained. How would one? I have never seen the hardware you have or held it in my hands.

HTH,
Patrick