OPNsense Forum

English Forums => Hardware and Performance => Topic started by: DerKammi on November 02, 2022, 11:39:31 am

Title: Disable flow control Mellanox ConnectX 2 on boot
Post by: DerKammi on November 02, 2022, 11:39:31 am

Hope someone can help me out with turning of txpause and rxpause during boot on a Mellanox ConnectX 2 NIC.

My ISP is requesting to disable txpause and rxpause if you want to run a direct fiber into your firewall. By using the following command I can disable this on both interfaces:

Code: [Select]

ifconfig mlxeno -mediaopt txpause,rxpause
ifconfig mlxen1 -mediaopt txpause,rxpause

This is verified in the ifconfig output and on the dashboard.

However I can't figure out how to make this happen during boot via either tunables or a config file.

I tried a couple of different tunables but none of them are supported after I enter them.

Code: [Select]

hw.mlx4_en.0.fc
hw.mlxen0.fc.
hw.mlxen.0.fc

Also with hw changed in dev. I tested more, don't remenber them all :)

Just to be sure, the dmesg output:

Code: [Select]

mlx4_core0: <mlx4_core> mem 0xfe900000-0xfe9fffff,0xf8000000-0xf9ffffff irq 24 at device 0.0 on pci1
mlx4_core: Mellanox ConnectX core driver v3.7.1 (November 2021)
mlx4_core: Initializing mlx4_core
mlx4_core0: Unable to determine PCI device chain minimum BW
mlx4_en mlx4_core0: Activating port:1
mlxen0: Ethernet address: 00:02:c9:5a:05:2c
mlx4_en: mlx4_core0: Port 1: Using 4 TX rings
mlxen0: link state changed to DOWN
mlx4_en: mlx4_core0: Port 1: Using 8 RX rings
mlx4_en: mlxen0: Using 4 TX rings
mlx4_en: mlxen0: Using 8 RX rings
mlx4_en: mlxen0: Initializing port
mlx4_en mlx4_core0: Activating port:2
mlxen1: Ethernet address: 00:02:c9:5a:05:2d
mlx4_en: mlx4_core0: Port 2: Using 4 TX rings
mlxen1: link state changed to DOWN
mlx4_en: mlx4_core0: Port 2: Using 8 RX rings
mlx4_en: mlxen1: Using 4 TX rings
mlx4_en: mlxen1: Using 8 RX rings
mlx4_en: mlxen1: Initializing port
mlxen0: link state changed to UP
mlxen1: link state changed to UP


Title: Re: Disable flow control Mellanox ConnectX 2 on boot
Post by: yourfriendarmando on November 04, 2022, 07:53:35 am

It looks like there could be a tunable:

Also search for a tunable:
sysctl -a | grep -iE 'fc|flow|pause'

https://network.nvidia.com/sites/default/files/pdf/prod_software/Mellanox_FreeBSD_User_Manual_v3.4.1.pdf&ved=2ahUKEwjby7LS95P7AhVzjokEHXGxCy8QFnoECBkQAQ&usg=AOvVaw0x1fWqDQQFK1wIJFbIhNFH

Title: Re: Disable flow control Mellanox ConnectX 2 on boot
Post by: tokade on November 07, 2022, 08:45:13 pm

Otherwise you can put a script with ifconfig commands unter /usr/local/etc/rc.syshook.d/start

Franco also mentioned individual settings for interfaces, but I haven't found them yet in the GUI.

Kind regards
Torsten

Title: Re: Disable flow control Mellanox ConnectX 2 on boot
Post by: DerKammi on November 26, 2022, 12:58:38 pm

Finnaly had some time to play with the suggestions and came to the conclusion that there are no tuneables for these X2 cards in OPNSense/FreeBSD and all is done via ipconfig.

The pdf mentioned is for newer generation cards which uses a different driver.

In the end I found a pdf for de X2 cards which uses ipconfig command for rxpause,txpause settings and sysctl commands for the most other settings of the X2 cards.

https://usermanual.wiki/m/d86f82c808d6b799caa50e77b7e97b12495b5b4b89b64471678c8d9e6895eab3.pdf

So I resulted to Torsten his suggestion of using a script in /usr/local/etc/rc.syshook.d/start/99-fc

Code: [Select]

#!/bin/sh

ifconfig mlxen0 -mediaopt txpause,rxpause
ifconfig mlxen1 -mediaopt txpause,rxpause

which is working fine.

So no longer I need to enter these commands manually after boot.

Thanks for the help.