Kernel Parameters on Tunables not working.

[rafaelbs]

Hi,

I have been trying to do change a few parameters to improve performance (https://docs.opnsense.org/troubleshooting/performance.html).

After adding the following tunables like:
net.isr.bindthreads = 1
net.isr.maxthreads = -1
net.inet.rss.enabled = 1

and reboot, nothing happens. When searching with sysctl -a, for example, still net.inet.rss.enabled shows 0.

There are not any other equals parameters conflicting. I already tried to change the loader.conf.local, which also did not work.

Any ideas about what am I doing wrong?

Thanks a lot.

[meyergru]

How did you set the tuneables? Did you add them under "System: Settings: Tunables"?

[rafaelbs]

Yes. I did under System: Settings: Tunables, but nothing happens.

We are running last version, completely updated.

I already tried with /boot/loader.conf.local and did not work.

[meyergru]

That is virtually impossible. Did you apply the settings after creating the variables?

BTW: You should not edit configuration files by hand, as they will be overwritten by the OpnSense config. IDK if /boot/loader.conf.local is being honored. There is a folder /boot/loader.conf.d where files can be put.

That you can check: When you apply the tuneables settings, they should turn up in /boot/loader.conf ("fgrep net.isr.bindthreads /boot/loader.conf").

[jonny5]

There's one isr setting that I believe really enables the rest, see attached image too.

net.isr.dispatch: deferred

[pfry]

There's one isr setting that I believe really enables the rest, see attached image too.

net.isr.dispatch: deferred

Interesting. I'd figure "net.inet.rss.enabled" would be the... enabler. I have "direct" (the default), and actual dispatch is mostly "hybrid":

Code Select Expand

root@fw:/home/user # sysctl net.isr
net.isr.numthreads: 16
net.isr.maxprot: 16
net.isr.defaultqlimit: 256
net.isr.maxqlimit: 10240
net.isr.bindthreads: 1
net.isr.maxthreads: 16
net.isr.dispatch: direct
root@fw:/home/user # sysctl net.inet | grep rss
net.inet.rss.bucket_mapping: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
net.inet.rss.enabled: 1
net.inet.rss.debug: 0
net.inet.rss.basecpu: 0
net.inet.rss.buckets: 8
net.inet.rss.maxcpus: 64
net.inet.rss.ncpus: 16
net.inet.rss.maxbits: 7
net.inet.rss.mask: 7
net.inet.rss.bits: 3
net.inet.rss.hashalgo: 2
root@fw:/home/user # netstat -Q
Configuration:
Setting                        Current        Limit
Thread count                        16          16
Default queue limit                256        10240
Dispatch policy                direct          n/a
Threads bound to CPUs          enabled          n/a

Protocols:
Name  Proto QLimit Policy Dispatch Flags
ip        1  1024    cpu  hybrid  C--
igmp      2    256 source  default  ---
rtsock    3    256 source  default  ---
arp        4    256 source  default  ---
ether      5    256    cpu  direct  C--
ip6        6  1024    cpu  hybrid  C--
ip_direct    9    256    cpu  hybrid  C--
ip6_direct    10    256    cpu  hybrid  C--
[...]

I believe the dispatch takes a bit of time to settle, but it's been a while since I set it up.

[rafaelbs]

Just can't make it work.

Attached image shows how my tunables are set.

A couple sysctl outputs:

Code Select Expand


root@OPNsense:~ # sysctl net.isr
net.isr.numthreads: 1
net.isr.maxprot: 16
net.isr.defaultqlimit: 256
net.isr.maxqlimit: 10240
net.isr.bindthreads: 0
net.isr.maxthreads: 1
net.isr.dispatch: deferred

root@OPNsense:~ # sysctl net.inet.rss
net.inet.rss.bucket_mapping: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 8:8 9:9 10:10 11:11 12:12 13:13 14:14 15:15 16:16 17:17 18:18 19:19 20:20 21:21 22:22 23:23 24:24 25:25 26:26 27:27 28:28 29:29 30:30 31:31 32:32 33:33 34:34 35:35 36:36 37:37 38:38 39:39 40:40 41:41 42:42 43:43 44:44 45:45 46:46 47:47 48:48 49:49 50:50 51:51 52:52 53:53 54:54 55:55 56:56 57:57 58:58 59:59 60:60 61:61 62:62 63:63
net.inet.rss.enabled: 0
net.inet.rss.debug: 0
net.inet.rss.basecpu: 0
net.inet.rss.buckets: 64
net.inet.rss.maxcpus: 64
net.inet.rss.ncpus: 64
net.inet.rss.maxbits: 7
net.inet.rss.mask: 63
net.inet.rss.bits: 6
net.inet.rss.hashalgo: 2

As we can see, net.inet.rss.enabled is not the only parameter that is not working. The net.isr.maxthreads did not worked, but curiously net.isr.dispatch has successfully changed.

Any idea or logs I could check?

[Patrick M. Hausen]

Does your interface hardware support RSS?

[rafaelbs]

Yes it does.

It is a Mellanox MCX516A-CCAT.

Firmware is updated.

Dmesg does not show any error message.

Is there any test I could do to make sure there are not any hardware or driver issues?

Thanks a lot.
Rafael

[meyergru]

I do not think that the hardware is the problem. It seems that the settings just do not get transferred to the *.conf files for some reason. Did you check if the settings are present in /boot/loader.conf?

You probably misspelled the variables, had whitespace before or after because of a copy&paste error?

[rafaelbs]

I do not think that the hardware is the problem. It seems that the settings just do not get transferred to the *.conf files for some reason. Did you check if the settings are present in /boot/loader.conf?

You probably misspelled the variables, had whitespace before or after because of a copy&paste error?

I had just checked, and parameters are correctly present on /boot/loader.conf, just like the print of Tunables I sent earlier.

I had also double checked the incidence of misspell or whitespaces in the variables and values, and there are none.

[meyergru]

But net.inet.rss.enabled is not hardware-dependent, so if that is is present in /boot/loader.conf, I cannot see why it is not visible via "sysctl net.inet.rss.enabled" - unless something else is missing.

I have set these values:

Code Select Expand

net.isr.bindthreads = 1
net.isr.maxthreads = -1
net.inet.rss.enabled = 1
net.inet.rss.bits = 2
net.isr.dispatch: deferred

Anything else that might be influential? Like virtualisation, number of cores = 1, Zenarmor, netmap?

[rafaelbs]

I'm not using virtualization and Zenarmor.

Number of cores and Netmap, what/where could I check?

My values are pretty much the same as yours, except rss.bits I'm using 6.

[meyergru]

You have 64 threads? Oh, yes. Judging from this, you even have 128. IDK if QAT changes anything in that.

Maybe you should try with a lower number, IDK if FreeBSD has problems with such high numbers.

[pfry]

[...]
My values are pretty much the same as yours, except rss.bits I'm using 6.

How many cores do you have? 6 bits suggests 64 (ah - just read the latest...). But the value doesn't have to be based on cores - it's apparently more of an entropy setting.

How about "netstat -Q"? I don't know that you'd need to post it; I truncated it above.