Broadcom P225P driver - vlans no longer work

[a.rueter]

Hi,

I installed the update from 23.1.11_1 to 23.7 on one of our firewalls yesterday and had problems with the broadcom P225P network card driver.

The problem only occurs when vlans are assigned, normal operation without vlans is no problem.
Result:

Code Select Expand

bnxt0: HWRM_CFA_L2_FILTER_ALLOC command returned INVALID_Params error

In the meantime I have the firewall running again on 23.1.11_1.

[franco]

Oh boy, there have been a number of changes in bnxt(4) from FreeBSD 13.1 to 13.2 and it's relatively clear that something was broken as per your report.

This may be one of these https://reviews.freebsd.org/D36443 but I don't see an easy way of reverting. There is probably an oversight somewhere in the refactor(s).

Best report this to https://bugs.freebsd.org/ directly.


Cheers,
Franco

[Patrick M. Hausen]

As a hack can you try enabling promiscuous mode on the Broadcom parent interface?

[a.rueter]

As a hack can you try enabling promiscuous mode on the Broadcom parent interface?

Unfortunately, that didn't work either. The bad thing is that I can't get to any interface apart from the broadcom network card either, as this somehow blocks everything.

Here is also a screenshot of the boot process where you can see these errors as an attachment.

Best report this to https://bugs.freebsd.org/ directly.

I have created a bug report:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272865

[franco]

Thanks, subscribed.

I can provide a test kernel as soon as a patch is proposed.


Cheers,
Franco

[SeMe1991]

Hello everyone, I just registered for this thread. I have the exact same problem. I was in our data center last night to exchange our old Opnsense (Mellanox Connect X3) for a new one with a total of 6 Broadcom cards 25Gbit (bnxt). I deliberately waited for V23.07 because FreeBSD 13.2 contains the latest drivers. All interfaces without a VLAN work as expected. However, our WAN uplink to the core routers must be transferred with VLAN. We found on the core routers that no packets are sent out at all via the Broadcom card with VLAN.

Just for Info:
The next problem is the FRR plugin (tested with BGP). It seems like the permissions of the directories and config-files are not set during installation. But you can change the permissions manually.

Cheers  :)

[Patrick M. Hausen]

We were hit by what is probably related, although our adapters are all BCM57416 10G.

The related pr seems to be:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133

I already contacted Chandrakanth but did not hear back, yet.

[SeMe1991]

To be exact:

My cards are 1x PCIe Quadport BCM57504 and 1x OCP3.0 Dualport BCM57412 (HP labeled)

[Thomas Niedermeier]

Hello everyone,

we internally tested also all of our Broadcom cards and the issue affects not all NICs.
Only the cards with the BCM574xx chip are affected from this, we also wrote a Wiki article about that issue: https://www.thomas-krenn.com/en/wiki/Broadcom_BCM574xx_VLAN_problem_under_FreeBSD_13.2_with_bnxt_driver
We are also in contact with Broadcom and Kevin Bowling already made a patch: https://reviews.freebsd.org/D41558

Best regards,
Thomas

[Werner Fischer]

Regarding NICs:

• Affected: NICs with Broadcom BCM574xx (NXE / Wh+) Chips, e.g. Broadcom P225P
• Not affected: NICs with Broadcom BCM575xx (Thor) Chips, e.g. Broadcom P425g, P2100G

For an overview, which Chips are used in which NICs see https://www.thomas-krenn.com/de/wiki/Broadcom_Netzwerkkarten

Regarding Kevin Bowling's Patch: we currently do not know whether it fixes the problem. We have not compiled the patch. I assume other FreeBSD users might comment soon here in the Bugreport:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133#c34

[franco]

I'll publish a kernel build for you guys in half an hour.


Cheers,
Franco

[franco]

Sorry, took a little longer:

# opnsense-update -zkr 23.7.2-bnxt


Cheers,
Franco

[Thomas Niedermeier]

Hi Franco,

wow very nice, we'll test the Kernel with the cards and give you feedback.
Hopefully it solves the problem, would be fantastic.

Cheers,
Thomas

[Thomas Niedermeier]

First update: We installed the Kernel and the problem seems fixed, VLANs work!
The interface was accessible via ping. But we will go on and test the new Kernel further in the next days.
Would be fantastic if we already solved the problem with this Kernel.

[franco]

Ok good start. I'll report on the review when you give your final verdict, but I don't have much doubt about shipping this as is in a future kernel.


Cheers,
Franco