RSS with Intel NIC (eg. X710 or E810) - CARP / LACP flappings because of CPU0

[jzah]

Hi guys

we try to build a HA cluster pair with two XEON based servers. We have Intel E710 and a X810 cards in each server. Our actual test was with the X810 card based on 8x10Gb links (two QSFP 40G optics, which are divided into 4x10G each). So we do LACP on 8x10Gbps links.
As clients wo do have 6 servers with 2x10Gbps (LACP bonded on RHEL9). Between two server in the same subnet we are able to get 20Gbps throughput with iperf3.
We have enabled RSS based on the documentation, which seems to work fine. We see, that based on the tunable net.inet.rss.bits, that eg. if we use "2" it uses multiple (4) CPUs. So far so good.

Our problem is, if we have high load on CPU0  (which is possible to achieve with one iperf3 stream from one client) we see that LACP and CARP doesn't work anymore on FreeBSD. OPNsense completely stuck related to CARP and LACP, which triggeres immediate failovers and flapping between the two firewalls.
We tried then to introduce the following iflib tunables: dev.ice.0.iflib.core_offset="1" (of course for all ice NICs). However it's still possible to see high load due to RSS on CPU0. So it doesn't work as expect.

How can we ensure with an Intel NIC that RSS doesn't use CPU0? Because it seems that the slow protocols like ARP/CARP/LACP are seems to be bound to CPU0....

On Chelsio cards there seems to be some tunables to fix that, eg. hw.cxgbe.rsrv_noflowq="1" for TX direction

We really stuck here, we experimented with cpuset as well, but didn't come to a working solution. Any help is really appreciated...

Cheers

[jzah]

Just for other people who are looking for a solution. It seems that the following helps to prevent RSS using CPU0...

Code: [Select]

# show cpu assignment to thread netisr for CPU0 -> eg. 100241
procstat -a -S
 
# set CPU2 to process netisr from CPU0 (of course in our test we had multiple cores available and RSS only enabled on CPU0/CPU1. So CPU2 was just idleing.
cpuset -l 2 -t 100241
With this setting RSS doesn't use anymore CPU0 and hence no drops on CPU0 anymore for CARP.