Messages

Hello,

we recently tested upgrades and also fresh installs of OPNsense 24.7 RC2 with different systems.
It runs fine without any issues hardware wise with our test systems.

We discovered that the bnxt driver for Broadcom 10G/25G NIC cards will be automatically loaded when such a card ist installed. Thats very nice and saves work during install!
It's already useable if you fire up a OPNsense 24.7 RC2 boot drive and run it in live mode.

But unfortunately I didn't find any hint proving this outcome. Do you know if the bnxt driver is now directly compiled into the kernel?
I did some research on a system with 24.7 RC2 and if you call kldstat it prompts that the if_bnxt.ko is loaded.

Best regards and thanks again for your great work!
Thomas

Oh okay sorry for that, I maybe got it wrong while reading your text initially.
Hmm yes mobile connections are always a bit buggy, because a LTE Modem (for example Quectel EM12-G) is kind of a computer on its own.

I recently tested it again, if you dare to unplug the SIM and insert it afterwards you might guess it would come up again and reconnect... no way you have to unplug and de-energize the whole machine and then boot it up again to get it working again.

The fact that you have to delete it completely... I have no clue...

What I can say is that if you stick to this article it works fine: https://docs.opnsense.org/manual/how-tos/multiwan.html
We also did our own tests and created a wiki article: https://www.thomas-krenn.com/de/wiki/OPNsense_Multi_WAN
I hope that helps.

Best regards,
Thomas

A move to FreeBSD 14 would be really great, because we discovered that now with the FreeBSD 14-Release (released a couple of days ago) the Atom Board with C5315 from Supermicro works just fine (A3SPI-4C-LN6PF). The same board we tested with a earlier version of FreeBSD 14 didn't boot at all (as I already replied in this thread).
So we thought let's give it a try and rerun the test with the current FreeBSD 14 version, as the test was nearly half a year ago. And as mentioned above, absolutely fine and no issues until now.

So it would be amazing if OPNsense would upgrade to a FreeBSD 14 Kernel next year!

Cheers,
Thomas

Here for confirmation a excerpt of "dmesg -a":

Code Select Expand


---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-RELEASE #0 releng/14.0-n265380-f9716eee8ab4: Fri Nov 10 05:57:23 UTC 2023
    root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
VT(efifb): resolution 1024x768
CPU: Intel Atom(R) C5315 processor (2400.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x80667  Family=0x6  Model=0x86  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4ff8ebff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2394b2c3<FSGSBASE,TSCADJ,FDPEXC,SMEP,ERMS,PQM,NFPUSG,PQE,RDSEED,SMAP,CLFLUSHOPT,CLWB,PROCTRACE,SHA>
  Structured Extended Features2=0x1a400124<UMIP,WAITPKG,GFNI,RDPID,CLDEMOTE,MOVDIRI,MOVDIR64B>
  Structured Extended Features3=0xfc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x20c6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16228294656 (15476 MB)
Event timer "LAPIC" quality 600


We did some further tests and the tests so far went well.
We already did some tests with ping and currently we are currently running iperf3 tests.
No issues so far. We are not done yet, but I wanted to keep you updated.

I also don't have much doubt about shipping this patch with a future kernel, would be great.

Cheers,
Thomas

First update: We installed the Kernel and the problem seems fixed, VLANs work!
The interface was accessible via ping. But we will go on and test the new Kernel further in the next days.
Would be fantastic if we already solved the problem with this Kernel.

Hi Franco,

wow very nice, we'll test the Kernel with the cards and give you feedback.
Hopefully it solves the problem, would be fantastic.

Cheers,
Thomas

Hello everyone,

we internally tested also all of our Broadcom cards and the issue affects not all NICs.
Only the cards with the BCM574xx chip are affected from this, we also wrote a Wiki article about that issue: https://www.thomas-krenn.com/en/wiki/Broadcom_BCM574xx_VLAN_problem_under_FreeBSD_13.2_with_bnxt_driver
We are also in contact with Broadcom and Kevin Bowling already made a patch: https://reviews.freebsd.org/D41558

Best regards,
Thomas

We tested the new kernel, works fine!
Driver is updated to 2.20.0.1. We did some quick tests, connected bnxt2 (first NIC on the P425G) to the network and it has received an address via dhcp.

root@OPNsense:~ # sysctl -a | grep -E 'dev.*.iflib.driver_version' -A 5
dev.bnxt.2.iflib.driver_version: 2.20.0.1
dev.bnxt.2.%domain: 0
dev.bnxt.2.%parent: pci10
dev.bnxt.2.%pnpinfo: vendor=0x14e4 device=0x1751 subvendor=0x14e4 subdevice=0x4250 class=0x020000
dev.bnxt.2.%location: slot=0 function=0 dbsf=pci0:1:0:0
dev.bnxt.2.%driver: bnxt

Quad Port P425G 25G Card (# sysctl -a |grep -i broadcom):

bnxt2: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc030000-0x303dbc03ffff,0x303dbb000000-0x303dbbffffff,0x303dbc058000-0x303dbc05ffff irq 88 at device 0.0 numa-domain 0 on pci10
bnxt3: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc020000-0x303dbc02ffff,0x303dba000000-0x303dbaffffff,0x303dbc050000-0x303dbc057fff irq 89 at device 0.1 numa-domain 0 on pci10
bnxt4: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc010000-0x303dbc01ffff,0x303db9000000-0x303db9ffffff,0x303dbc048000-0x303dbc04ffff irq 88 at device 0.2 numa-domain 0 on pci10
bnxt5: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc000000-0x303dbc00ffff,0x303db8000000-0x303db8ffffff,0x303dbc040000-0x303dbc047fff irq 89 at device 0.3 numa-domain 0 on pci10

Dual Port 100G P2100G Card:

bnxt6: <Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbf010000-0x303dbf01ffff,0x303dbe000000-0x303dbeffffff,0x303dbf028000-0x303dbf02ffff irq 90 at device 0.0 numa-domain 0 on pci11
bnxt7: <Broadcom BCM57508 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbf000000-0x303dbf00ffff,0x303dbd000000-0x303dbdffffff,0x303dbf020000-0x303dbf027fff irq 91 at device 0.1 numa-domain 0 on pci11

Hi Franco,

thanks for the new sets!
I will test it tomorrow with the Broadcom cards and give feedback.

Best regards,
Thomas

Hello,

yes these Broadcom Cards don't work with OPNsense 23.1, we discovered this issue internally some time ago.
The FreeBSD driver for these quite new cards is too old to support them, but Broadcom updated it and with FreeBSD 13.2-Release these cards are recognised and work quite good.
I only tested it with plain FreeBSD 13.2-Release, because OPNsense is based on 13.1-Release.
But Franco tweeted in May that he successfully booted OPNsense with 13.2, so I hope future OPNsense 23.7 is based on 13.2, that would fix the issue with these cards.

Best regards,
Thomas

Output from FreeBSD 13.2-Release:

root@:~ # sysctl -a |grep -i broadcom

bnxt2: <Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet> mem 0x303dbc030000-0x303dbc03ffff,0x303dbb000000-0x303dbbffffff,0x303dbc058000-0x303dbc05ffff irq 88 at device 0.0 numa-domain 0 on pci11

--> BCM57504 is the IC of the P425G

New driver version in FreeBSD 13.2-Release:
root@:~ # sysctl -a | grep -E 'dev.*.iflib.driver_version' -A 5
dev.bnxt.7.iflib.driver_version: 2.20.0.1
dev.bnxt.7.%domain: 0
dev.bnxt.7.%parent: pci12
dev.bnxt.7.%pnpinfo: vendor=0x14e4 device=0x1750 subvendor=0x14e4 subdevice=0x2100 class=0x020000
dev.bnxt.7.%location: slot=0 function=1 dbsf=pci0:2:0:1
dev.bnxt.7.%driver: bnxt

Hi,

we are currently evaluating a Supermicro A3SPI-4C-LN6PF motherboard with OPNsense. Fails also with error 19, so I came across this message. Tried to install FreeBSD, similar issue "Mounting from ufs:/dev/ufs/FreeBSD_Install failed with errror 19."Also tried the FreeBSD 14.0 Current code. Tried USB 2.0 flash drives, IPMI ISO mount, DVD drives. Always the same error. Seems that the architecture is not supported yet. We already reported the problem to Supermicro.

Best regards,
Thomas

Hi Franco,

I did the tests for Werner and you'll find the output of dmesg and installer.log attached to my post.
I hope these two files help to fix the problem.

Cheers,
Thomas