Messages

I just installed pfsense to test...

Its working there, without enabling anything...

Hi,

I recently purchased a R86S-N305C which has two 25G interfaces from a MLX Connectx-4 card.

The issue that I'm facing is that it does not seem to negotioate correctly, I worked at my own ISP so I know what they have configured and I know it is correct.

Normally to get 25Gbit/s WAN Speeds we did set the Autoneg to off, Speed to 25Gbit/s and FEC to ON. I cannot see any documentations on how to do that except this thread with no answers on it.  https://forum.opnsense.org/index.php?topic=30467.msg146919#msg146919

Could anyone help me config wise to turn FEC on? Or maybe something more temporary than described in the above post?

Best regards

Drivers getting loaded:

Code Select Expand

root@OPNsense:~ # dmesg | grep mlx
mlx5_core0: <mlx5_core> mem 0x60e2000000-0x60e3ffffff at device 0.0 on pci4
mlx5: Mellanox Core driver 3.7.1 (November 2021)uhub1: 2 ports with 2 removable, self powered
mlx5_core0: INFO: mlx5_port_module_event:710:(pid 12): Module 0, status: unplugged
mlx5_core0: WARN: mlx5_vsc_set_space:125:(pid 0): Space 0x7 is not supported.
mlx5_core0: WARN: mlx5_fwdump_prep:102:(pid 0): VSC scan space is not supported
mlx5_core: INFO: (mlx5_core0): E-Switch: Total vports 9, l2 table size(65536), per vport: max uc(1024) max mc(16384)
mlx5_core1: <mlx5_core> mem 0x60e0000000-0x60e1ffffff at device 0.1 on pci4
mlx5_core1: INFO: mlx5_port_module_event:705:(pid 12): Module 1, status: plugged and enabled
mlx5_core1: WARN: mlx5_vsc_set_space:125:(pid 0): Space 0x7 is not supported.
mlx5_core1: WARN: mlx5_fwdump_prep:102:(pid 0): VSC scan space is not supported
mlx5_core: INFO: (mlx5_core1): E-Switch: Total vports 9, l2 table size(65536), per vport: max uc(1024) max mc(16384)
mlx5_core0: ERR: mlx5_cmd_check:710:(pid 0): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x6c4d48)
mlx5_core1: ERR: mlx5_cmd_check:710:(pid 0): ACCESS_REG(0x805) op_mod(0x1) failed, status bad parameter(0x3), syndrome (0x6c4d48)
mce0: ERR: mlx5e_ioctl:3514:(pid 40442): tso4 disabled due to -txcsum.
mce0: ERR: mlx5e_ioctl:3527:(pid 41160): tso6 disabled due to -txcsum6.
mce1: ERR: mlx5e_ioctl:3514:(pid 43036): tso4 disabled due to -txcsum.
mce1: ERR: mlx5e_ioctl:3527:(pid 43090): tso6 disabled due to -txcsum6.

IFCONFIG

Code Select Expand

mce0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN (wan)
        options=7e8800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO>
        ether 24:8a:07:95:d9:18
        inet6 fe80::268a:7ff:fe95:d918%mce0 prefixlen 64 tentative scopeid 0x8
        media: Ethernet autoselect <rxpause,txpause>
        status: no carrier (Cable is unplugged)
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

Also mlxen5 yes is active in /boot/loader

I just did, am I correct on just sending an E-Mail to that address?

This issue is driving me nuts lol

Hi the Card is an CNA X710-DA2 Manufactured 07/2018.

Yes it seems like I may have to compile the drivers myself.

Also it may be some issue with PCI Passthrough but I don't think thats the case since as stated it didnt work on Bare-Metal either.

Would you be able to assist me in Compiling them?

I also upgraded de Firmware of the Card to the newest version, didnt help.

Bare Metal install also throws the same errors, so its definitely not related to proxmox or the PCI passthrough

Yes I'm using Proxmox.

But it still seems the driver does not load correctly. Still getting the same error both in pfsense and opnsense. Will try to do a bare metal install now...

Hi everyone

I'm currently facing a really weird issue that I cannot find any info of in the internet  :-\

So I've just set up PCI passthrough, and added both of them to the OPNsense VM. So far so good. Everything there seems to work. IOMMU is working fine.

The WAN Interface also gets an IP so traffic is flowing there.

The LAN Interface is where my problem is; it has IP 192.168.1.1, its connected to a Mikrotik switch that has no VLANs and traffic over it is running fine.

But I cannot access the OPNsense Webinterface nor ping it.

The WAN Interface can ping for example 8.8.8.8 perfectly.

Now the errors I'm getting:

ixl1 (LAN Interface) i40e_aq_add_macvlan status I40E_ERR_ADMIN_QUEUE_ERROR, error I40E_AQ_RC_BAD_ADDR

I cannot seem to find any info about this anywhere, does someone of you know how I can fix this?

Edit: now both interfaces seem to get these error messages and I've lost the IP of my ISP...
Do I need drivers?

Edit 2: I think OPNsense 23.7 doesn't have ixl support for X710?

Hi everyone

I have a Proxmox Server with a 6 Core 12 Thread 4.7GhZ AMD Ryzen 5 7600X and 32GB DDR5 RAM. In that Proxmox Server I have virtualised an opnsense that does NAT, DHCP and inter-vlan Routing.

My Problem is that I only get around 2.5 - 3 Gbit/s on my 10Gbit/s interface. It is connected directly to my ISP where I work at as a Network Engineer, however as I work at an ISP I do not work with Firewalls that much because they live at our customers.

PCI-E is fast enough and has enough lanes. CPU should be fast enough as well.

I don't know if I'm missing any tunables or options or is there something else I can check?

Edit: Sorry I'm using the Intel X710-DA2  not the Mellanox.

Edit 2: Maybe something I could try would be PCI Passthrough into the VM that is running opnsense. But I would rather not do that because it involves a lot of work again.