Mellanox ConnectX-3 support

[cyrus104]

I was able to follow mimugmail's instructions that were posted to the forum and to the site at the bottom of this post. I was able to get the driver to autoload on boot and updated the firmware.

These cards are able to do 56Gbps instead of just 40Gbps if being used with an FDR cable and capable switch such as the SX6036 I'm using. I have this card in some ESXi machines and everything is detected as 56Gbps as expected.

I would like to see if there is a way to set these cards at the faster speed in OPNsense, the interface settings only offer autoselect, 1G, 10G, and 40G right now.

I would also like to see if there are specific options to enable OPNsense to take "full" advantage of these cards such as hardware offloading. I am using VLANs so, the hardware offloading might not work.

Any help would be greatly appreciated.

https://www.routerperformance.net/opnsense/mellanox-connecx-management-in-opnsense/

[mimugmail]

Please dont use offloading, it will break couple of features.

How much throughput so you achieve?
With Chelsio I got 26Gbit max

[cyrus104]

I haven't gotten far enough along in rebuilding my network to get a full iperf using the 56gbps cards. I have 1 installed in my OPNsense box. I'm installing them in my 3 ESXi machines and a TrueNAS box today. That should give me a much better understanding of their performance.

Part of my issues was not knowing the link speed that the OPNsense actually negotiated to.

I wanted to see what you would put for all of the hardware offloading options. I uploaded a picture of my config, I think I've disabled all offloading. I left VLAN hardware filtering on but can turn that off if you recommend it.

The iperfs that I have done are from other machines with 10Gbps which all pull around 9.4Gbps if I run: iperf -c -p -P 20

Thanks for the quick response.

[mimugmail]

It shouldnt influence the Performance that much when disabled

[cyrus104]

That's good to hear. I now have them all disabled.

Any idea how to get the interface to detect the 56gbps speed?

[mimugmail]

No idea, maybe ask at freebsd-net mailing list, there are mellanox devs around, maybe some sysctl tweaks