Messages

1

Hardware and Performance / Mellanox ConnectX-2 Infiniband Modules

« on: July 10, 2020, 09:05:12 pm »

I am attempting to use a Mellanox ConnectX-2 MHQH29B-XTR card in Infiniband mode. It is not available as an interface to be added in the GUI.

I am new to FreeBSD & OPNsense.

Code: [Select]

root@OPNsense:~ # uname -r
11.2-RELEASE-p16-HBSD

Code: [Select]

pciconf -lv
none3@pci0:2:0:0:       class=0x0c0600 card=0x003515b3 chip=0x673c15b3 rev=0xb0 hdr=0x00
    vendor     = 'Mellanox Technologies'
    device     = 'MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE]'
    class      = serial bus
I attempted to install the modules from Mellanox as follows & resulting errors:

Code: [Select]

root@OPNsense:~/MLNX_OFED_FREEBSD_V2.1.5/modules/mlx4 # make -m /usr/src/share/mk SYSDIR=/usr/src/sys clean cleandepend
make: "/root/MLNX_OFED_FREEBSD_V2.1.5/modules/mlx4/Makefile" line 16: Malformed conditional (${MK_INET_SUPPORT} != "no")
make: "/root/MLNX_OFED_FREEBSD_V2.1.5/modules/mlx4/Makefile" line 21: Malformed conditional (${MK_INET6_SUPPORT} != "no")
make: Fatal errors encountered -- cannot continue
make: stopped in /root/MLNX_OFED_FREEBSD_V2.1.5/modules/mlx4
I tried copying in modules compiled for FreeBSD 10.1 from the internet & not surprisingly received version mismatched errors.

I looked at this post but was unable to get the script to work using echo:
https://forum.opnsense.org/index.php?topic=12140.msg55610#msg55610

I believe I did use the above linked method to get the kernel sources.

I attempted to compile the modules per the 1st script from this website:
https://wiki.freebsd.org/InfiniBand

Code: [Select]

root@OPNsense:~ # ./buildInstallLoadDriversReformat.sh 4 datagram
There were several warnings for this & other packed member names:

Code: [Select]

warning: taking address of packed member 'hdr' of class or structure 'cm_req_msg' may result in an
      unaligned pointer value [-Waddress-of-packed-member]
At the end of the compilation was the following:

Code: [Select]

kldxref /boot/modules
kldload: an error occurred while loading the module. Please check dmesg(8) for more details.
./buildInstallLoadDriversReformat.sh: auto-append-line: not found
Id Refs Address            Size     Name
 1   32 0x0 22847d8  kernel
 2    1 0x0 e248     carp.ko
 3    1 0x0 e4e8     if_bridge.ko
 4    2 0x0 6cb0     bridgestp.ko
 5    1 0x0 3250     if_enc.ko
 6    1 0x0 6e30     if_gre.ko
 7    1 0x0 14788    if_lagg.ko
 8    1 0x0 7bb8     if_tap.ko
 9    3 0x0 55050    pf.ko
10    1 0x0 2ab8     pflog.ko
11    1 0x0 d590     pfsync.ko
12    1 0x0 6acd0    mlx4.ko
13    2 0x0 265d8    linuxkpi.ko
16    1 0x0 6fc4     tmpfs.ko
17    1 0x0 7298     aesni.ko

To disable Infiniband support, remove or comment out the following device
loads in /boot/loader.conf:

mlx4 ibcore mlx4ib ipoib

Code: [Select]

root@OPNsense:~ # dmesg
link_elf_obj: symbol vnet_entry_if_index undefined
linker_load_file: Unsupported file type
link_elf_obj: symbol vnet_entry_if_index undefined
linker_load_file: Unsupported file type
KLD mlx4ib.ko: depends on ibcore - not available or version mismatch
linker_load_file: Unsupported file type
link_elf_obj: symbol vnet_entry_if_index undefined
linker_load_file: Unsupported file type
KLD ipoib.ko: depends on ibcore - not available or version mismatch
linker_load_file: Unsupported file type

Code: [Select]

root@OPNsense:/boot/modules # ls
ibcore.ko       ipoib.ko        linker.hints    mlx4.ko         mlx4ib.ko

Code: [Select]

root@OPNsense:~ # kldload /boot/modules/ibcore.ko
kldload: an error occurred while loading the module. Please check dmesg(8) for more details.

Code: [Select]

root@OPNsense:~ # dmesg
link_elf_obj: symbol vnet_entry_if_index undefined
linker_load_file: Unsupported file type
The interface does not show up in the GUI.

I added system tunables to load the modules at boot, but that doesn't work probably because I can't load the modules at all.

I tried compiling the modules with the same script on a FreeBSD 11.2 virtual machine & received the same errors.

This forum post also mentioned some of the same errors, but it didn't seem like a solution was reached:
https://hup.hu/node/166600

I am at the end of my understanding as to how to solve this, any help would be much appreciated.

Would compiling the kernel help me toward this goal, either in OPNsense or a VM? I looked at the Infiniband FreeBSD wiki & it mentioned it, but I became confused as to how & where settings were & worked.
https://wiki.freebsd.org/InfiniBand