I did some RSS testing when I was on 21.7.5 and there was no discernible impact so I left the tunables in when I upgraded to 22.7. I later saw some of the tunables labelled as unsupported so I removed from the UI. All good.
I am tracing a marked reduction in bitrate from WAN to LAN which made me look at tunables again and the status of RSS as part of the elimination process. Cables and similar are part of the process at the moment of course, and in the meantime, this question.
BEFORE:
I had OPN 21.7.5, OpenSSL, Hardware is APU4.
Upgraded BIOS beforehand, coreboot v 4.14.0.6 .
Network interfaces on this system are igb.
penguin@OPNsense:~ % sudo sysctl -a | grep -i 'isr.bindthreads\|isr.maxthreads\|inet.rss.enabled\|inet.rss.bits'
net.inet.rss.enabled: 1
net.inet.rss.bits: 2
net.isr.bindthreads: 1
net.isr.maxthreads: 4
penguin@OPNsense:~ % sudo netstat -Q
Configuration:
Setting Current Limit
Thread count 4 4
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs enabled n/a
Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 cpu hybrid C--
igmp 2 256 source default ---
rtsock 3 256 source default ---
arp 4 256 source default ---
ether 5 256 cpu direct C--
ip6 6 256 cpu hybrid C--
ip_direct 9 256 cpu hybrid C--
ip6_direct 10 256 cpu hybrid C--
NOW:
Now on OPN 22.7.4, OpenSSL, same hardware.
root@OPNsense:/home/penguin # sysctl -a | grep -i 'isr.bindthreads\|isr.maxthreads\|inet.rss.enabled\|inet.rss.bits'
net.inet.rss.enabled: 0
net.inet.rss.bits: 2
net.isr.bindthreads: 0
net.isr.maxthreads: 1
root@OPNsense:/home/penguin # netstat -Q
Configuration:
Setting Current Limit
Thread count 1 1
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs disabled n/a
Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 flow default ---
igmp 2 256 source default ---
rtsock 3 256 source default ---
arp 4 256 source default ---
ether 5 256 source direct ---
ip6 6 1000 flow default ---
So it looks like during the upgrade, RSS was disabled. Question 1: Is that expected?
Since I realised that today, I've re-added the tunables that are accepted "net.inet.rss.bits", "net.isr.maxthreads" and "net.inet.rss.enabled" and rebooted . I also had to but netstat -Q shows that threads aren't bound to CPU enabled.
Configuration:
Setting Current Limit
Thread count 4 4
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs disabled n/a
Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 cpu hybrid C--
igmp 2 256 source default ---
rtsock 3 256 source default ---
arp 4 256 source default ---
ether 5 256 cpu direct C--
ip6 6 1000 cpu hybrid C--
ip_direct 9 256 cpu hybrid C--
ip6_direct 10 256 cpu hybrid C--
Question 2: Should I expect the setting to change ie. is wrong at present without threads bound to CPU?
Question 3: How do I make it so?
Thanks for the assistance.
It says "NOW" but
net.inet.rss.enabled: 0
Is this correct?
Cheers,
Franco
Quote from: cookiemonster on September 21, 2022, 06:18:15 PM
Since I realised that today, I've re-added the tunables that are accepted "net.inet.rss.bits", "net.isr.maxthreads" and "net.inet.rss.enabled" and rebooted . I also had to but netstat -Q shows that threads aren't bound to CPU enabled.
Question 2: Should I expect the setting to change ie. is wrong at present without threads bound to CPU?
Question 3: How do I make it so?
I'm missing the "net.inet.rss.bindthreads" tunable from your list of applied steps. Maybe you forgot it?
That was the question Franco. It was enabled, then system upgraded, then disabled. The question was if it was expected for the upgrades to retain or remove the rss enablement.
Since I found it disabled, I've gone and set again to 1 via tunable.
The second question is if "Threads bound to CPUs disabled n/a" is correct and expected. If not, how do I re-enable it.
Quote from: tuto2 on September 26, 2022, 11:22:46 AM
Quote from: cookiemonster on September 21, 2022, 06:18:15 PM
Since I realised that today, I've re-added the tunables that are accepted "net.inet.rss.bits", "net.isr.maxthreads" and "net.inet.rss.enabled" and rebooted . I also had to but netstat -Q shows that threads aren't bound to CPU enabled.
Question 2: Should I expect the setting to change ie. is wrong at present without threads bound to CPU?
Question 3: How do I make it so?
I'm missing the "net.inet.rss.bindthreads" tunable from your list of applied steps. Maybe you forgot it?
Thanks Tuto2. I did try and just did now but I get "unsupported" appear once added in the tunables. Let me see if I can find a name change in sysctl list.
root@OPNsense:/home/penguin # sysctl -a | grep bind
kern.ipc.tls.bind_threads: 1
net.isr.bindthreads: 0
user.posix2_c_bind: 0
So I've tried adding the tunable as net.isr.bindthreads and net.inet.rss.bindthreads
Both give an "unsupported" message.
Is that expected? Which is right for this kernel version? 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 stable/22.7-n250229-c8a7fd0d5dc SMP amd64
Correction.
I've used the net.isr.bindthreads tunable now instead. I'm guessing net.inet.rss.bindthreads was replaced?
I need to reboot later and check again if it sticks.
Quote from: cookiemonster on September 26, 2022, 11:32:02 AM
So I've tried adding the tunable as net.isr.bindthreads and net.inet.rss.bindthreads
Both give an "unsupported" message.
Is that expected? Which is right for this kernel version? 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 stable/22.7-n250229-c8a7fd0d5dc SMP amd64
My mistake, it should be "net.isr.bindthreads" (not net.isr.rss.bindthreads).
After applying and rebooting:
Configuration:
Setting Current Limit
Thread count 1 1
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs enabled n/a
I see no indication of unsupported anywhere.
Which "unsupported" message you mean? In the log for runtime settings? The GUI says "boot-time" for net.inet.rss.enabled which is correct...
Cheers,
Franco
It's been spotted what the problem was franco, thanks. It should be "net.isr.bindthreads" (not net.isr.rss.bindthreads). The wrong one caused the "unsupported" message on the UI when adding it.
In light of this if the tunable string changed between 22.1 and 22.7, and I found it gone, I wonder if the upgrade routines should have re-enabled the new one if the old one was enabled at upgrade time.
For now I just need to reboot and I'll be golden :)
I think it was always net.isr.bindthreads as it pertains to more than just RSS, maybe some old forum post got this wrong? The other one likely did not work in any version of the RSS patching.
Cheers,
Franco
OK. It probably is that then. Reboot done. All appears good again. I'll mark it as solved.
Thanks Franco and Tuto2 for your help as always.
yay :)