OPNsense Forum
English Forums => 23.7 Legacy Series => Topic started by: a.rueter on August 01, 2023, 12:43:01 pm
-
Hi,
I installed the update from 23.1.11_1 to 23.7 on one of our firewalls yesterday and had problems with the broadcom P225P network card driver.
The problem only occurs when vlans are assigned, normal operation without vlans is no problem.
Result:
bnxt0: HWRM_CFA_L2_FILTER_ALLOC command returned INVALID_Params error
In the meantime I have the firewall running again on 23.1.11_1.
-
Oh boy, there have been a number of changes in bnxt(4) from FreeBSD 13.1 to 13.2 and it's relatively clear that something was broken as per your report.
This may be one of these https://reviews.freebsd.org/D36443 but I don't see an easy way of reverting. There is probably an oversight somewhere in the refactor(s).
Best report this to https://bugs.freebsd.org/ directly.
Cheers,
Franco
-
As a hack can you try enabling promiscuous mode on the Broadcom parent interface?
-
As a hack can you try enabling promiscuous mode on the Broadcom parent interface?
Unfortunately, that didn't work either. The bad thing is that I can't get to any interface apart from the broadcom network card either, as this somehow blocks everything.
Here is also a screenshot of the boot process where you can see these errors as an attachment.
Best report this to https://bugs.freebsd.org/ directly.
I have created a bug report:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272865
-
Thanks, subscribed.
I can provide a test kernel as soon as a patch is proposed.
Cheers,
Franco
-
Hello everyone, I just registered for this thread. I have the exact same problem. I was in our data center last night to exchange our old Opnsense (Mellanox Connect X3) for a new one with a total of 6 Broadcom cards 25Gbit (bnxt). I deliberately waited for V23.07 because FreeBSD 13.2 contains the latest drivers. All interfaces without a VLAN work as expected. However, our WAN uplink to the core routers must be transferred with VLAN. We found on the core routers that no packets are sent out at all via the Broadcom card with VLAN.
Just for Info:
The next problem is the FRR plugin (tested with BGP). It seems like the permissions of the directories and config-files are not set during installation. But you can change the permissions manually.
Cheers :)
-
We were hit by what is probably related, although our adapters are all BCM57416 10G.
The related pr seems to be:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133
I already contacted Chandrakanth but did not hear back, yet.
-
To be exact:
My cards are 1x PCIe Quadport BCM57504 and 1x OCP3.0 Dualport BCM57412 (HP labeled)
-
Hello everyone,
we internally tested also all of our Broadcom cards and the issue affects not all NICs.
Only the cards with the BCM574xx chip are affected from this, we also wrote a Wiki article about that issue: https://www.thomas-krenn.com/en/wiki/Broadcom_BCM574xx_VLAN_problem_under_FreeBSD_13.2_with_bnxt_driver
We are also in contact with Broadcom and Kevin Bowling already made a patch: https://reviews.freebsd.org/D41558
Best regards,
Thomas
-
Regarding NICs:
- Affected: NICs with Broadcom BCM574xx (NXE / Wh+) Chips, e.g. Broadcom P225P
- Not affected: NICs with Broadcom BCM575xx (Thor) Chips, e.g. Broadcom P425g, P2100G
For an overview, which Chips are used in which NICs see https://www.thomas-krenn.com/de/wiki/Broadcom_Netzwerkkarten
Regarding Kevin Bowling's Patch: we currently do not know whether it fixes the problem. We have not compiled the patch. I assume other FreeBSD users might comment soon here in the Bugreport:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133#c34
-
I'll publish a kernel build for you guys in half an hour.
Cheers,
Franco
-
Sorry, took a little longer:
# opnsense-update -zkr 23.7.2-bnxt
Cheers,
Franco
-
Hi Franco,
wow very nice, we'll test the Kernel with the cards and give you feedback.
Hopefully it solves the problem, would be fantastic.
Cheers,
Thomas
-
First update: We installed the Kernel and the problem seems fixed, VLANs work!
The interface was accessible via ping. But we will go on and test the new Kernel further in the next days.
Would be fantastic if we already solved the problem with this Kernel.
-
Ok good start. I'll report on the review when you give your final verdict, but I don't have much doubt about shipping this as is in a future kernel.
Cheers,
Franco
-
We did some further tests and the tests so far went well.
We already did some tests with ping and currently we are currently running iperf3 tests.
No issues so far. We are not done yet, but I wanted to keep you updated.
I also don't have much doubt about shipping this patch with a future kernel, would be great.
Cheers,
Thomas
-
It landed in FreeBSD main branch and I've also queued it up for next kernel in 23.7.x unless you tell me I should remove it.
Cheers,
Franco
-
It landed in FreeBSD main branch and I've also queued it up for next kernel in 23.7.x unless you tell me I should remove it.
Cheers,
Franco
Thats really good to hear. I have a (hopefully simple) question regarding upgrading to this:
I have a hardware installation with only affected NICs which leads to a broken system once upgrading to 23.7. How do I upgrade from 23.1.11_1 directly to a minor version containing the fixed kernel? or will there be a fixed 23.7 base kernel the 23.7 upgrade will use, that can be further upgraded to 23.7.x?
Regards,
Rudolf
-
The situation is a bit tricky here. I'd recommend staying on 23.1.x until 23.7.x is fixed. I think it's going to be 23.7.4 which is a little longer, but once that happens I can move the 23.1.11 upgrade to 23.7.4 directly so people don't run into it.
And for full context: 23.7.3 is this week but will not offer a new kernel for other reasons.
Cheers,
Franco
-
Hi Franco,
I think there is an important comment here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269133#c47 with a commit that should be maybe included in 23.7.4, too:
bnxt: Don't restart on VLAN changes
In rS360398, a new iflib device method was added with default of opt out
for VLAN events needing an interface reset.
This is unintentional for bnxt(4) and is causing another bug in its VLAN
initialization code to affect the common case of adding and removing
VLANs on an existing interface.
What do you think?
Best regards, Werner
-
Same commit for stable/13. I already have it on our stable/23.7 and will release this week. Likely on Thursday.
Cheers,
Franco
-
Same commit for stable/13. I already have it on our stable/23.7 and will release this week. Likely on Thursday.
Cheers,
Franco
Is the direct upgrade path from 23.1 to 23.7 on systems with only affected NICs now fixed as well? or is there anything to note or do for a successful upgrade to 23.7.4+ ?
-
Same commit for stable/13. I already have it on our stable/23.7 and will release this week. Likely on Thursday.
Cheers,
Franco
Is the direct upgrade path from 23.1 to 23.7 on systems with only affected NICs now fixed as well? or is there anything to note or do for a successful upgrade to 23.7.4+ ?
To answer my own question: Yes it fixed, upon upgrading to 23.7 it will directly upgrade to 23.7.5
-
Sorry I was not in the office last week. Indeed we previously fixed the kernel due to this issue on upgrades and yesterday I changed the whole upgrade path to 23.7.5 (base and kernel being at 23.7.4 which is correct).
Cheers,
Franco