mlx4en: CQE completed in Error

[SirUffsALot]

Hello, I have a strange problem.

For performance reasons, I switched from a Sophos XG105 to a Lenovo M720Q ThinClient (Pentium G5400T, 16GB RAM).
Initially, a Mellanox ConnectX-2 card was installed, but I replaced it relatively quickly with a Mellanox ConnectX-3 card (HP 546SFP+).

However, I have the following phenomena with both cards (At first i thought that the old one was broken):


Sometimes after a reboot (e.g. after updates), network connections are unusably slow. If I then connect a monitor, I see that there are kernel messages spamming with the following content (also see attached image):
"mlx4_en: mlxen0: CQE completed in error - vendor syndrom: 0xf9 syndrom: 0x5"
If I then restart the firewall via the opnsense menu, the error disappears in 75% of cases and everything runs as desired (even for weeks) - if not, a second reboot usually helps.

My setup consists of a single interface (but the plan is actually to use an LACP) with various VLANs. One of them with PPPoE.
Hardware CRC, TSO, LRO and VLAN Filtering is enabled (did not change the default) and i don't think that's the cause, because the problem is not persistent.

BIOS is up-to-date. Tried 23.7 and 24.1, both affected.

any ideas where i could start?

[SirUffsALot]

Well, it seems that the hardware offload was the culprit after all.

I changed the setup to a vm with sr-iov under proxmox and had the same problem when I imported the old configuration.
With a fresh install (where the offload features were disabled) the problem disappeared and i can still hit about ~6gbit routing, which is enough for me.

Case closed.