Poor 10Gb performance, need help troubleshooting

[rungekutta]

To add 2 more points.

First, adding

Code Select Expand


hw.cxgbe.nrxq_vi=8
hw.cxgbe.ntxq_vi=8
hw.cxgbe.nnmtxq_vi=8
hw.cxgbe.nnmrxq_vi=8


creates 8 rx/tx queues also for these virtual ports:

Code Select Expand


vcxl0: netmap queues/slots: TX 8/1023, RX 8/1024
vcxl0: 8 txq, 8 rxq (NIC); 8 txq, 8 rxq (netmap)
vcxl1: netmap queues/slots: TX 8/1023, RX 8/1024
vcxl1: 8 txq, 8 rxq (NIC); 8 txq, 8 rxq (netmap)


Second, unfortunately, Suricata now destroys performance, even without any rules active!

Here's a WAN speed test from another machine (through OPNsense)

Code Select Expand


root@xxx:~/tmp # ./fast
-> 984.61 Mbps
root@xxx:~/tmp # ./fast
-> 5.55 Gbps


First is with Suricata enabled and in IPS mode (but no rules), the second is with Suricata disabled. Disappointing. But maybe this can be tuned. For now, I'm disabling Suricata.

[testo_cz]

@rungekutta
Very nice info about your NIC setup. And throughput results.

Was the earlier  firmware in the NIC something like too old or it was perhaps customized ?
Because as people often reuse HW / NICs , it might not have a genuin firmware. For example customized by e.g. server vendor.

I'm only getting familiar with Suricata.... Is it like utilizing 100% CPU if enabled ?

[rungekutta]

Was the earlier  firmware in the NIC something like too old or it was perhaps customized ?
Because as people often reuse HW / NICs , it might not have a genuin firmware. For example customized by e.g. server vendor.

The firmware was pretty old, 1.12 something which according to release notes is from 2014. Current version is 1.26 and from 2021. So Proxmox updated this for me, but not the boot rom, which was also from 2014. So when I had managed to update that as well (with Chelsio tools in DOS, booting from a USB stick) I could pass through the card ok to the OpnSense VM.

I'm only getting familiar with Suricata.... Is it like utilizing 100% CPU if enabled ?

When I had rules enabled it pegged something like 3 or 4 cores (out of 8 available) but never got above 1Gb/s. Without any rules it used less CPU (less than 1 core) but still limited throughput to roughly 1Gb/s. I haven't looked much into tuning it, but there aren't many exposed options either via the GUI.