iommu & QLogic NetXtreme II BCM57840

[aab]

Hi,
I have virtualized the opnsense and pass-through QLogic NetXtreme II BCM57840  via iommu.
I can't get both the ports work, one of them works but the other has the problem with the address mapping. I am just wondering if someone else has faced the same problem, the logs are as follow, please guide !
If I insert the card in the other PCI-express slot the bxe0 starts working and bxe1 start having the problem which is mentioned below for bxe0

Code Select Expand

bxe0: <QLogic NetXtreme II BCM57840 4x10GbE (B1) BXE v:1.78.91> mem 0x380000000000-0x3800007fffff,0x380000800000-0x380000ffffff,0x380002000000-0x38000200ffff irq 10 at device 9.0 on pci0
bxe0: PCI BAR0 [10] memory allocated: 0x380000000000-0x3800007fffff (8388608) -> 0xfffffe008d400000
bxe0: PCI BAR2 [18] memory allocated: 0x380000800000-0x380000ffffff (8388608) -> 0xfffffe008dc00000
bxe0: PCI BAR4 [20] memory allocated: 0x380002000000-0x38000200ffff (65536) -> 0xfffffe0084678000
bxe0: ERROR: Invalid SHMEM validity signature: 0x00010008
bxe0: ERROR: Invalid phy config in NVRAM (PHY1=0x00010008 PHY2=0x00010008)
bxe0: Unknown media!
bxe0: IFMEDIA flags : 20
bxe0: Using defaults for TSO: 65518/35/2048
bxe0: ERROR: FW failed to respond!
bxe0: ERROR: MCP response failure, aborting
bxe0: ERROR: Failed to unload previous driver! time_counter 10 rc -1
bxe1: <QLogic NetXtreme II BCM57840 4x10GbE (B1) BXE v:1.78.91> mem 0x380001000000-0x3800017fffff,0x380001800000-0x380001ffffff,0x380002010000-0x38000201ffff irq 10 at device 10.0 on pci0
bxe1: PCI BAR0 [10] memory allocated: 0x380001000000-0x3800017fffff (8388608) -> 0xfffffe008e800000
bxe1: PCI BAR2 [18] memory allocated: 0x380001800000-0x380001ffffff (8388608) -> 0xfffffe008f000000
bxe1: PCI BAR4 [20] memory allocated: 0x380002010000-0x38000201ffff (65536) -> 0xfffffe0084688000
bxe1: Found 10Gb Fiber media.
bxe1: IFMEDIA flags : 20
bxe1: Using defaults for TSO: 65518/35/2048
bxe1: Ethernet address: d4:5d:64:09:0b:6a
bxe1: MSI-X vectors Requested 2 and Allocated 2

Any ideas ?

[netnut]

Any ideas ?

As a dual port NIC, it probably consists of _two_ PCI devices, did you map _both_ PCI id's ?

Are those two PCI devices in the same IOMMU group ?

Code Select Expand

find /sys/kernel/iommu_groups/ -type l

[aab]

Hi again,
Thanks for the reply :)
lspci -nnk shows following for the 10G cards, both have the same PCI_ids.

Code Select Expand

61:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
Subsystem: ASUSTeK Computer Inc. Device [1043:8591]
Kernel driver in use: vfio-pci
Kernel modules: bnx2x
61:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
Subsystem: ASUSTeK Computer Inc. Device [1043:8591]
Kernel driver in use: vfio-pci
Kernel modules: bnx2x

[Patrick M. Hausen]

lspci -nnk shows following for the 10G cards, both have the same PCI_ids.

Nope. One has ID 61:00.0, the other one 61:00.1 - you need to map both explicitly in your VM/hypervisor config.

[cookiemonster]

easiest is to follow the well known instructions used for pcie passthrough for graphics cards to VMs.

[aab]

Since both the ports are on the same card, their ID is same i.e. 14e4:16a1 and 1043:8591, please check following

Code Select Expand

61:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
Subsystem: ASUSTeK Computer Inc. Device [1043:8591]
Kernel driver in use: vfio-pci
Kernel modules: bnx2x
61:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
Subsystem: ASUSTeK Computer Inc. Device [1043:8591]
Kernel driver in use: vfio-pci
Kernel modules: bnx2x

My grub has following parameters,

Code Select Expand

rhgb quiet intel_iommu=on iommu=pt pci-stub.ids=1043:8591


Following files exists with the data as shown

Code Select Expand

[root@server2]# cat /etc/modules-load.d/vfio-pci.conf
vfio-pci
[root@server2]# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=14e4:16a1
[root@server2]#



I have the same configuration on two servers, the servers are a little different, the 10G cards are the same, on server2 irqbypass is missing as shown below. I don't remember what did I do with server1 to get it to work.

Code Select Expand

[root@server1 etc]# lsmod | grep kvm
kvm_intel             442368  0
kvm                  1347584  1 kvm_intel
irqbypass              16384  2 vfio_pci_core,kvm    ##   << this isn't shown on server2

[root@server2 etc]# lsmod | grep kvm
kvm_intel             442368  0
kvm                  1347584  1 kvm_intel

[netnut]

I have the same configuration on two servers, the servers are a little different

And because they're a little different, they're not the same...

If you don't specify what the difference is, you basically starting a guessing game: I go for the 3 points, server 2 has a >v6.2 kernel...

You asked for ideas, I asked you to c/p your IOMMU groups, somehow you decided it's not relevamt ?!?!

What type of motherboard and what kind of PCI slot is this card in ? That is, what are the actual/physical lanes the PCI slot provides, something you have to look up in the specs of your motherboard, the physical slot itself says nothing about the lanes. You could have a x16 slot with only 4 lanes attached.

So my next guess, now for 5 points, your x8 card is in a x4 slot...

[aab]

The server (Server1) on which both ports work, PCI-E slot is

Code Select Expand

Type:  PCIExpressGen4

And the server (Server2) on which one port works the PCI-E slot is

Code Select Expand

Type:  PCIExpressGen3
:o

[cookiemonster]

Since both the ports are on the same card, their ID is same i.e. 14e4:16a1 and 1043:8591, please check following

Code Select Expand

61:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
Subsystem: ASUSTeK Computer Inc. Device [1043:8591]
Kernel driver in use: vfio-pci
Kernel modules: bnx2x
61:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
Subsystem: ASUSTeK Computer Inc. Device [1043:8591]
Kernel driver in use: vfio-pci
Kernel modules: bnx2x

I don't think you can pass both through in this way, since as you well point out, the pci id are the same. Only one.
Why not add the _whole_ pci device to the VM though? In other words, without vfio.
Is it Proxmox?

[aab]

Since both the ports are on the same card, their ID is same i.e. 14e4:16a1 and 1043:8591, please check following

Code Select Expand

61:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
Subsystem: ASUSTeK Computer Inc. Device [1043:8591]
Kernel driver in use: vfio-pci
Kernel modules: bnx2x
61:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
Subsystem: ASUSTeK Computer Inc. Device [1043:8591]
Kernel driver in use: vfio-pci
Kernel modules: bnx2x

I don't think you can pass both through in this way, since as you well point out, the pci id are the same. Only one.
Why not add the _whole_ pci device to the VM though? In other words, without vfio.
Is it Proxmox?

How to passthrough without vfio ? it is opnsense and the vm-host is centos/kvm

[Patrick M. Hausen]

That's what I intended to suggest: pass both 61:00.0 and 61:00.1 as PCI(e) devices.

[cookiemonster]

Problem here is that 61:00.0 and 61:00.1 are not the PCIe device identifiers Patrick. They are the bus IDs and you can only to my knowledge, tell the hypervisor to pass through something, by the PCIe ID. But they are the same one in this case.

> How to passthrough without vfio ? it is opnsense and the vm-host is centos/kvm
That I don't know. Proxmox allows you to use the whole device. I'll have to check later if you don't get to it first.

I usually find the Arch docs quite good for when I need to figure out my Proxmox problems as a sounding board https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF

[Patrick M. Hausen]

Problem here is that 61:00.0 and 61:00.1 are not the PCIe device identifiers Patrick. They are the bus IDs and you can only to my knowledge, tell the hypervisor to pass through something, by the PCIe ID. But they are the same one in this case.

The two hypervisors I am familiar with - bhyve and ESXi - use bus ID, so I assumed this was the general approach. Apologies if that assumption is wrong and my conclusion misleading.

[cookiemonster]

Not at all. All perspectives are useful.
I also acknowledge that I have been wrong. It seems the bus id can be used on vfio.
OP, see:
https://forum.proxmox.com/threads/pass-through-1-hba-but-not-the-other-when-they-both-have-the-same-device-id.117542/
and
https://forum.proxmox.com/threads/pci-passthrough-selection-with-identical-devices.63042/#post-287937