Messages

We have 4 cards MCX516A-CCAT, and each has 2 100Gbps ports, and also 2 on-board gigabit ethernet.
[...]

Are the two 1GbEs 1000BASE-T, perhaps i210s, with one attached to a BMC? Probably not an interface/driver issue, then. I'd give the buckets a shot - it seems the Mellanox has an RSS mapping limit (not surprising). I don't know if this would disable RSS, but it can't hurt to poke it. You might check for complaints from the 1GbEs, too, and pick a safe start value.

Only one 1GbE is connected, but not enabled. There is a dedicated IPMI port.

Here are logs from the 1GbEs:

Code Select Expand

[1] igb0: <Intel(R) I350 (Copper)> port 0xd000-0xd01f mem 0x9a620000-0x9a63ffff,0x9a644000-0x9a647fff at device 0.0 on pci21
[1] igb0: EEPROM V1.63-0 eTrack 0x800009fa
[1] igb0: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb0: Using 8 RX queues 8 TX queues
[1] igb0: Using MSI-X interrupts with 9 vectors
[1] igb0: netmap queues/slots: TX 8/1024, RX 8/1024
[1] igb1: <Intel(R) I350 (Copper)> port 0xd020-0xd03f mem 0x9a600000-0x9a61ffff,0x9a640000-0x9a643fff at device 0.1 on pci21
[1] igb1: EEPROM V1.63-0 eTrack 0x800009fa
[1] igb1: Using 1024 TX descriptors and 1024 RX descriptors
[1] igb1: Using 8 RX queues 8 TX queues
[1] igb1: Using MSI-X interrupts with 9 vectors
[1] igb1: netmap queues/slots: TX 8/1024, RX 8/1024
[199] igb0: link state changed to UP


We have 4 cards MCX516A-CCAT, and each has 2 100Gbps ports, and also 2 on-board gigabit ethernet.

From the 8 100Gbps ports, we are using 2, one for LAN e other to WAN

From the 2 1Gbps ports, we are using 1 for backup WAN link.

Since the beginning, all NICs were working by default, firewall is working fine. The challenge here is performance increasing to work with Suricata on IPS mode.

After a couple tests, I figured that Tunables that I set on GUI (System / Settings / Tunables) are correcting appearing on /boot/loader.conf but they don't take any effect.

Here are the messages from dmesg related to my interface:

Code Select Expand

[2] mlx5_core0: <mlx5_core> mem 0x4007e000000-0x4007fffffff at device 0.0 on pci1
[2] mlx5: Mellanox Core driver 3.7.1 (November 2021)ahciem0: Unsupported enclosure interface
...

[18] mlx5_core3: <mlx5_core> mem 0x3007c000000-0x3007dffffff at device 0.1 on pci6
[18] mlx5_core3: INFO: mlx5_port_module_event:709:(pid 12): Module 1, status: plugged and enabled
[18] mlx5_core3: INFO: health_watchdog:577:(pid 0): PCIe slot advertised sufficient power (75W).
[23] mlx5_core3: INFO: init_one:1713:(pid 0): cannot find SR-IOV PCIe cap
[23] mlx5_core: INFO: (mlx5_core3): E-Switch: Total vports 1, l2 table size(65536), per vport: max uc(128) max mc(2048)
[23] mlx5_core3: Failed to initialize SR-IOV support, error 2

...

[213] mce3: ERR: mlx5e_ioctl:3608:(pid 19542): tso4 disabled due to -txcsum.
[213] mce3: ERR: mlx5e_ioctl:3621:(pid 19776): tso6 disabled due to -txcsum6.

....

[214] mce3: INFO: mlx5e_open_locked:3297:(pid 0): NOTE: There are more RSS buckets(64) than channels(61) available


Any idea if those messages are related to get net.inet.rss.enabled on?

Thanks.
Rafael

[...]
My values are pretty much the same as yours, except rss.bits I'm using 6.

How many cores do you have? 6 bits suggests 64 (ah - just read the latest...). But the value doesn't have to be based on cores - it's apparently more of an entropy setting.

How about "netstat -Q"? I don't know that you'd need to post it; I truncated it above.

Here is my netstat -Q

Code Select Expand

# netstat -Q
Configuration:
Setting                        Current        Limit
Thread count                         1            1
Default queue limit                256        10240
Dispatch policy               deferred          n/a
Threads bound to CPUs         disabled          n/a

Protocols:
Name   Proto QLimit Policy Dispatch Flags
ip         1   1000   flow  default   ---
igmp       2    256 source  default   ---
rtsock     3    256 source  default   ---
arp        4    256 source  default   ---
ether      5    256 source   direct   ---
ip6        6   1000   flow  default   ---

Workstreams:
WSID CPU   Name     Len WMark   Disp'd  HDisp'd   QDrops   Queued  Handled
   0   0   ip         0    24        0        0        0   518021   518021
   0   0   igmp       0     0        0        0        0        0        0
   0   0   rtsock     0     1        0        0        0       17       17
   0   0   arp        0    56        0        0        0   489663   489663
   0   0   ether      0     0   522772        0        0        0   522772
   0   0   ip6        0     3        0        0        0     1534     1534


You have 64 threads? Oh, yes. Judging from this, you even have 128. IDK if QAT changes anything in that.

Maybe you should try with a lower number, IDK if FreeBSD has problems with such high numbers.

Yep, we have 64 cores and 128 threads.

And we already tried with a lower number, without success.

I'm not using virtualization and Zenarmor.

Number of cores and Netmap, what/where could I check?

My values are pretty much the same as yours, except rss.bits I'm using 6.

I do not think that the hardware is the problem. It seems that the settings just do not get transferred to the *.conf files for some reason. Did you check if the settings are present in /boot/loader.conf?

You probably misspelled the variables, had whitespace before or after because of a copy&paste error?

I had just checked, and parameters are correctly present on /boot/loader.conf, just like the print of Tunables I sent earlier.

I had also double checked the incidence of misspell or whitespaces in the variables and values, and there are none.

Yes it does.

It is a Mellanox MCX516A-CCAT.

Firmware is updated.

Dmesg does not show any error message.

Is there any test I could do to make sure there are not any hardware or driver issues?

Thanks a lot.
Rafael

Just can't make it work.

Attached image shows how my tunables are set.

A couple sysctl outputs:

Code Select Expand


root@OPNsense:~ # sysctl net.isr
net.isr.numthreads: 1
net.isr.maxprot: 16
net.isr.defaultqlimit: 256
net.isr.maxqlimit: 10240
net.isr.bindthreads: 0
net.isr.maxthreads: 1
net.isr.dispatch: deferred

root@OPNsense:~ # sysctl net.inet.rss
net.inet.rss.bucket_mapping: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 8:8 9:9 10:10 11:11 12:12 13:13 14:14 15:15 16:16 17:17 18:18 19:19 20:20 21:21 22:22 23:23 24:24 25:25 26:26 27:27 28:28 29:29 30:30 31:31 32:32 33:33 34:34 35:35 36:36 37:37 38:38 39:39 40:40 41:41 42:42 43:43 44:44 45:45 46:46 47:47 48:48 49:49 50:50 51:51 52:52 53:53 54:54 55:55 56:56 57:57 58:58 59:59 60:60 61:61 62:62 63:63
net.inet.rss.enabled: 0
net.inet.rss.debug: 0
net.inet.rss.basecpu: 0
net.inet.rss.buckets: 64
net.inet.rss.maxcpus: 64
net.inet.rss.ncpus: 64
net.inet.rss.maxbits: 7
net.inet.rss.mask: 63
net.inet.rss.bits: 6
net.inet.rss.hashalgo: 2

As we can see, net.inet.rss.enabled is not the only parameter that is not working. The net.isr.maxthreads did not worked, but curiously net.isr.dispatch has successfully changed.

Any idea or logs I could check?

Yes. I did under System: Settings: Tunables, but nothing happens.

We are running last version, completely updated.

I already tried with /boot/loader.conf.local and did not work.

Hi,

I have been trying to do change a few parameters to improve performance (https://docs.opnsense.org/troubleshooting/performance.html).

After adding the following tunables like:
net.isr.bindthreads = 1
net.isr.maxthreads = -1
net.inet.rss.enabled = 1

and reboot, nothing happens. When searching with sysctl -a, for example, still net.inet.rss.enabled shows 0.

There are not any other equals parameters conflicting. I already tried to change the loader.conf.local, which also did not work.

Any ideas about what am I doing wrong?

Thanks a lot.

Hi,

I'm trying to improve my IPS performance (LAN only). Problem appears when I activate IPS: cpu ok (less than 10%), but a lot of network users can't use internet, and bandwidth decreases a lot.

After a lot of reading on forum and docs, I decided to follow the instructions on this article: https://docs.opnsense.org/troubleshooting/performance.html

After reboot, I realized that net.inet.rss.enabled is still 0.

If I'm not wrong, this is the key to allow more queues on NIC (100Gb).

I also have tried to increase the number of threads of Suricata, without success. Runmode is as Workers, and Threads (on netmap) auto. Changed threads to 16, but did not take any effect. It only starts with 2 threads.

Code Select Expand

<Notice> -- Threads created -> W: 2 FM: 1 FR: 1   Engine started.
Running last update of OPN (OPNsense 25.7.5-amd64).

Appreciate any help.

Regards.
Rafael

Hi!

We are planning to setup Transparent Web Proxy to a big network (around 10K hosts).

Our appliance spec is:

- AMD EPYC 7H12 64-Core Processor (64 cores, 128 threads)
- 524Gb of RAM
- 8 Nvidias
- Also has in Intel Quick Assist 8970.

I would like to hear your opinion if this spec is OK to run all the web traffic. On peak the box (without Proxy) is processing around 30Gbps of throughput, which most is web traffic.

In case not, I would like to know if is possible to setup a pool of opnsense instances specifically to run Web Proxy under a load balancer (HAproxy), and if there is a know guide to do it.


Thanks a lot!

Solution:

Since IPSEC did not work, even reinstalling, and we did not figure out what was going on, the only option was to rollback to Community (which is running perfectly), and wait for BE 25.10 that hopefully will work fine.


Thanks for all replies. 

One of your suggestions yesterday was related to a similar issue, where I saw the alternative of manually load IPSEC on 20-modules file.

I have just removed it and reboot the box. Ipsec did not start automatically.

Firewall general logs shows:

2025-10-10T11:48:04-03:00   Notice   kernel   [201] KLD ipsec.ko: depends on kernel - not available or version mismatch   
2025-10-10T11:48:04-03:00   Notice   kernel   [201] KLD ipsec.ko: depends on kernel - not available or version mismatch   
2025-10-10T11:48:04-03:00   Notice   kernel   [201] KLD ipsec.ko: depends on kernel - not available or version mismatch   
2025-10-10T11:48:04-03:00   Notice   opnsense-business   /usr/local/sbin/pluginctl: plugins_configure route_reload (execute task : system_routing_configure(1,[]))   
2025-10-10T11:48:04-03:00   Notice   opnsense-business   /usr/local/sbin/pluginctl: plugins_configure route_reload (1,[])   
2025-10-10T11:48:04-03:00   Notice   root   /usr/local/etc/rc.d/strongswan: WARNING: failed precmd routine for strongswan   
2025-10-10T11:48:04-03:00   Notice   root   /usr/local/etc/rc.d/strongswan: WARNING: Unable to load kernel module ipsec   
2025-10-10T11:48:04-03:00   Notice   kernel   [200] KLD ipsec.ko: depends on kernel - not available or version mismatch