Hey, all. I tried enabling RSS on my Supermicro box with ixl 10gb interfaces. Following the instructions, I ran netstat -Q, but still see Dispatch policy = direct. Everything else seems to look correct. Thoughts? Thanks!
I set these four tunables:
+ <item>
+ <tunable>net.isr.bindthreads</tunable>
+ <value>1</value>
+ <descr>Set manually to enable RSS</descr>
+ </item>
+ <item>
+ <tunable>net.isr.maxthreads</tunable>
+ <value>-1</value>
+ <descr>Set manually to enable RSS</descr>
+ </item>
+ <item>
+ <tunable>net.inet.rss.enabled</tunable>
+ <value>1</value>
+ <descr>Set manually to enable RSS</descr>
+ </item>
+ <item>
+ <tunable>net.inet.rss.bits</tunable>
+ <value>3</value>
+ <descr>Set manually to enable RSS</descr>
+ </item>
When I run netstat -Q, I get:
Configuration:
Setting Current Limit
Thread count 8 8
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs enabled n/a
Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 cpu hybrid C--
igmp 2 256 source default ---
rtsock 3 256 source default ---
arp 4 256 source default ---
ether 5 256 cpu direct C--
ip6 6 1000 cpu hybrid C--
ip_direct 9 256 cpu hybrid C--
ip6_direct 10 256 cpu hybrid C--
Workstreams:
WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled
0 0 ip 0 9 0 19585 0 2482 22067
0 0 igmp 0 0 24 0 0 0 24
0 0 rtsock 0 0 0 0 0 0 0
0 0 arp 0 0 388 0 0 0 388
0 0 ether 0 0 97182 0 0 0 97182
0 0 ip6 0 0 0 0 0 0 0
0 0 ip_direct 0 0 0 0 0 0 0
0 0 ip6_direct 0 0 0 0 0 0 0
1 1 ip 0 25 0 0 0 20988 20988
1 1 igmp 0 0 0 0 0 0 0
1 1 rtsock 0 2 0 0 0 107 107
1 1 arp 0 0 0 0 0 0 0
1 1 ether 0 0 0 0 0 0 0
1 1 ip6 0 0 0 0 0 0 0
1 1 ip_direct 0 0 0 0 0 0 0
1 1 ip6_direct 0 0 0 0 0 0 0
2 2 ip 0 32 0 0 0 12189 12189
2 2 igmp 0 0 0 0 0 0 0
2 2 rtsock 0 0 0 0 0 0 0
2 2 arp 0 0 0 0 0 0 0
2 2 ether 0 0 31153 0 0 0 31153
2 2 ip6 0 0 0 0 0 0 0
2 2 ip_direct 0 0 0 0 0 0 0
2 2 ip6_direct 0 0 0 0 0 0 0
3 3 ip 0 157 0 6561 0 94986 101547
3 3 igmp 0 0 0 0 0 0 0
3 3 rtsock 0 0 0 0 0 0 0
3 3 arp 0 0 0 0 0 0 0
3 3 ether 0 0 0 0 0 0 0
3 3 ip6 0 0 0 0 0 0 0
3 3 ip_direct 0 0 0 0 0 0 0
3 3 ip6_direct 0 0 0 0 0 0 0
4 4 ip 0 47 0 0 0 9834 9834
4 4 igmp 0 0 0 0 0 0 0
4 4 rtsock 0 0 0 0 0 0 0
4 4 arp 0 0 0 0 0 0 0
4 4 ether 0 0 37183 0 0 0 37183
4 4 ip6 0 0 0 0 0 0 0
4 4 ip_direct 0 0 0 0 0 0 0
4 4 ip6_direct 0 0 0 0 0 0 0
5 5 ip 0 16 0 125 0 8996 9121
5 5 igmp 0 0 0 0 0 0 0
5 5 rtsock 0 0 0 0 0 0 0
5 5 arp 0 0 0 0 0 0 0
5 5 ether 0 0 0 0 0 0 0
5 5 ip6 0 0 0 0 0 0 0
5 5 ip_direct 0 0 0 0 0 0 0
5 5 ip6_direct 0 0 0 0 0 0 0
6 6 ip 0 65 0 0 0 16411 16411
6 6 igmp 0 0 0 0 0 0 0
6 6 rtsock 0 0 0 0 0 0 0
6 6 arp 0 0 0 0 0 0 0
6 6 ether 0 0 111034 0 0 0 111034
6 6 ip6 0 0 0 0 0 0 0
6 6 ip_direct 0 0 0 0 0 0 0
6 6 ip6_direct 0 0 0 0 0 0 0
7 7 ip 0 50 0 0 0 47801 47801
7 7 igmp 0 0 0 0 0 0 0
7 7 rtsock 0 0 0 0 0 0 0
7 7 arp 0 0 0 0 0 0 0
7 7 ether 0 0 0 0 0 0 0
7 7 ip6 0 0 0 0 0 0 0
7 7 ip_direct 0 0 0 0 0 0 0
7 7 ip6_direct 0 0 0 0 0 0 0
Running sysctl -a | grep rss returns:
net.inet.rss.bucket_mapping: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
net.inet.rss.enabled: 1
net.inet.rss.debug: 0
net.inet.rss.basecpu: 0
net.inet.rss.buckets: 8
net.inet.rss.maxcpus: 64
net.inet.rss.ncpus: 8
net.inet.rss.maxbits: 7
net.inet.rss.mask: 7
net.inet.rss.bits: 3
net.inet.rss.hashalgo: 2
hw.bxe.udp_rss: 0
hw.ix.enable_rss: 1
Well, after reading the code again, I see that Dispatch policy is still set to direct, but ip, ip6, ip_direct, and ip6_direct, under Protocols, are all set to hybrid. Was it correct the whole time? Is the Dispatch policy under Configuration the wrong thing to base my assumption on?
Yes. :)
This is what counts:
Name Proto QLimit Policy Dispatch Flags
ip 1 1000 cpu hybrid C--
ip6 6 1000 cpu hybrid C--
[...]
ip_direct 9 256 cpu hybrid C--
ip6_direct 10 256 cpu hybrid C--
Plus if you look at the workstreams you will see that you have 8 of each of them associated to every single CPU.
QuotePlus if you look at the workstreams you will see that you have 8 of each of them associated to every single CPU.
Got it. I was (obviously) a bit confused given that Dispatch policy under Configuration was still shown as direct. I see an opportunity to update the docs a bit. :)
Anyway, thanks so much! I appreciate the reply!
I think this best explains why we still see "direct" under Configuration -> Dispatch policy: https://forum.opnsense.org/index.php?topic=24409.msg118244#msg118244