[Tutorial/Call for Testing] Enabling Receive Side Scaling on OPNsense

[madj42]

Does anyone know if there is a way to change the policy on the ip6 protocol from flow to cpu?  That is the difference I notice in some of the previous posts.  Thinking this may be why I'm not getting RSS on ip6.

[hemirunner426]

Also decided to test this out with a Protectli FW6B.

- WAN is PPPoE 1G/1G
- LAN is split between LAN and 4 VLANs
- Sensei running on LAN.

I set net.inet.rss.bits = 2.  Not sure if this is correct on a 2 core/2HT processor.  Seems to work fine on speedtest with reasonable CPU usage.

Also something to keep in mind, if you upgrade to a hotfix release it will replace the RSS kernel.  Be sure to reinstall the RSS kernel if you upgrade to the hotfix released yesterday.

[franco]

> Also something to keep in mind, if you upgrade to a hotfix release it will replace the RSS kernel.  Be sure to reinstall the RSS kernel if you upgrade to the hotfix released yesterday.

That's why you can lock any package to prevent it from upgrading. ;)

[hemirunner426]

I found the lock in the UI.  That's handy.  ;D

[dinguz]

I am seeing something odd here. If I have RSS enabled, unbound is no longer working on all threads. I have a 2 core machine, unbound starts with 2 threads, but thread 0 is not doing any work. When I disable RSS using the sysctl and reboot, unbound is working normally again.
Are any other people seeing this?

I have reported this a few weeks ago, and I'm wondering if I'm the only one experiencing this. Are other people seeing this as well, or have been able to reproduce?

It's quite easy to check: just enable RSS, reboot, go to the Unbound stats page in the GUI, and you'll probably see all zeroes for thread 0.


BTW: how do I revert to the stock kernel? If I do

Code Select Expand

opnsense-update -kr 21.7.2 it keeps trying to download a RSS version of the kernel.

[madj42]

Same thing here with the kernel.  No matter what you type in, it will append -rss to the name.  Looking at the verbose output it's saying that it's trying to download from the sets folder but when I look on every mirror, these kernels are in the snapshots folder.  Not sure if they were removed?  Due to this I'm getting a no valid signature error.

[franco]

It's trying to stick to the kernel device designation which is correct historically speaking. We rarely do have different kernel builds and only really needed them with ARM which is where this feature comes from.

This should force it back to the generic device configuration (the -D '' bit is important)

# opnsense-update -kf -D ''


Cheers,
Franco

[franco]

PS: https://github.com/opnsense/tools/commit/6fa3d553a -- this only affects testing kernels really and no production kernel can exhibit this issue.

[dinguz]

Another observation, this is part of the 'netstat -Q' output from a stock kernel:

Code Select Expand


Protocols:
Name   Proto QLimit Policy Dispatch Flags
ip         1   1000   flow  default   ---
igmp       2    256 source  default   ---
rtsock     3    256 source  default   ---
arp        4    256 source  default   ---
ether      5    256 source   direct   ---
ip6        6    256   flow  default   ---



Below is the 'netstat -Q' output from a system with the RSS kernel, but RSS disabled (net.inet.rss.enabled=0):

Code Select Expand


Protocols:
Name   Proto QLimit Policy Dispatch Flags
ip         1   2048   flow  default   ---
igmp       2    256 source  default   ---
rtsock     3    256 source  default   ---
arp        4    256 source  default   ---
ether      5    256 source   direct   ---
ip6        6    256    cpu   hybrid   C--


Notice the difference for ip6? Apparently the sysctl does not completely enable/disable RSS for ip6.

[franco]

Let me check when I have some spare time. Stephan is not available at the moment to look into it.


Cheers,
Franco

[franco]

Stephan found it, small issue with a refactor from my side:

# opnsense-update -zfkr 21.7.2-rss

The sticky device issue was also fixed, the kernel should report:

% uname -v
FreeBSD 12.1-RELEASE-p20-HBSD  b9b6960472e(master) SMP


Cheers,
Franco

[dinguz]

Thanks for the quick reply. Has anyone been able to have a look at the interaction with unbound?

[MartB]

Stephan found it, small issue with a refactor from my side:

# opnsense-update -zfkr 21.7.2-rss

The sticky device issue was also fixed, the kernel should report:

% uname -v
FreeBSD 12.1-RELEASE-p20-HBSD  b9b6960472e(master) SMP


Cheers,
Franco

Not available to download, no signature found.

[franco]

Not available to download, no signature found.

Make sure:

1. Default mirror is set
2. "-z" is being used as indicated

It looks like everything is there: https://pkg.opnsense.org/FreeBSD:12:amd64/snapshots/sets/

I can look at Unbound issue too as soon as I can finish my work on the upcoming 21.10 business update.


Cheers,
Franco

[tuto2]

Thanks for the quick reply. Has anyone been able to have a look at the interaction with unbound?

I will take a look at it, but as Franco noted I have a limited amount of time at hand right now. Will update as soon as I can.

Cheers,

Stephan