Sensei stopped swap use greater than 30%

[naltalef]

Hello.
I'm testing Sensei but it stops after a while because it found a swap usage greater than 30%.

I am using version 1.7.1 with OPNSense 20.7.7_1

The box is a Lanner NCA-4210B with a Core i7 7100 processor and 16 GB of RAM.

Memory usage with Sensei turned ON is approx 45% and Swap use 0% (76/8192 MB).

Deployment mode: Pasive
Database: Local Elasticsearch
Deployment Size: Large

Could someone help me to track down the source of the issue ?
Any suggestions are greatly appreciated.

Regards
Norberto

[mb]

Hi @naltalef,

It's probably your system started swapping due to high memory consumption.

1.7.1 is a bit old. Any chances that you can upgrade to OPNsense 21.1.x and Sensei 1.8.2?
Sensei 1.8.2 tells you which processes are using the most memory and you can track down the source.

For a workaround, you can also increase your SWAP warning threshold via Sensei -> Configuration -> Updates & Health.
Find "Max Swap Utilization (% of total SWAP):" through the bottom of page. Since you're using the Passive mode, this should not have any negative implications.

[naltalef]

Hi MB. Many thanks for your answer.

Hi.

> It's probably your system started swapping due to high memory consumption.
Yes was what I also assumed. But let me point some details.

After 24h with Sensei ON the swap usage grew up from 0% to 15% and RAM from 45% to 65%.
I can't discover why is the swap used since there are no processes that use it.

Running top -S -W -o swap:

75 processes:  2 running, 72 sleeping, 1 waiting
CPU:  2.0% user,  0.0% nice,  1.2% system,  0.3% interrupt, 96.5% idle
Mem: 3381M Active, 3317M Inact, 5061M Laundry, 1823M Wired, 1278M Buf, 2290M Free
Swap: 8192M Total, 1261M Used, 6931M Free, 15% Inuse

  PID   USERNAME    THR PRI  NICE   SIZE    RES SWAP STATE    C   TIME    WCPU COMMAND
   11    root                   8     155 ki31      0       128K      0     RUN      0       ?        775.80% idle
80319 elasticsea        90     52    0      12G    2694M    0     uwait    7  73:37   2.94% java
    0     root                  37   -16    -           0        592K      0    swapin   3  28.0H   2.14% kernel
18726 root                 11     20  -20    2510M   967M    0      nanslp   2  31:53   2.02% eastpect
77943 root                 11     20  -20    2510M   926M    0     nanslp   1  28:37   1.99% eastpect
44833 root                 11     20  -20    3625M  2087M    0    nanslp   3  34:04   1.68% eastpect
58255 root                 11     20  -20    3625M  1959M    0    nanslp   4  29:39   1.56% eastpect


It is a basic OPNSense installation with the only peculiarity that it has a pair of interfaces defined as a bridge by a client's specific request. That bridge has an average traffic of about 8 mbps.
I have each of two bridge interfaces as protected in Sensei.

After stopped Sensei, swap goes down to 1% and RAM to 26%.
If I stopped Elasticserach also Swap remains in the same value and RAM goes down to 10%
After restarted both process RAM goes to 48% and swap keeps in 1%.
I suspect that tomorrow I will find the increased values again

Is Sensei using swap in any way?

> 1.7.1 is a bit old. Any chances that you can upgrade to OPNsense 21.1.x and Sensei 1.8.2?
> Sensei 1.8.2 tells you which processes are using the most memory and you can track down the source.

It's not easy for me to upgrade because I should disconnect the bridge interfaces while doing it. I am not close to where it is installed, however I could upgrade if you suspect it can help.

Do you think it is not related to Sensei? I can leave it off for a few days and see what happens.

> For a workaround, you can also increase your SWAP warning threshold via Sensei -> Configuration -> Updates & Health.
> Find "Max Swap Utilization (% of total SWAP):" through the bottom of page. Since you're using the Passive mode, this should not > have any negative implications.

I saw this option. However, if swap grows every day, would it always end up giving the error or not?

I really appreciate your advice.

Regards
Norberto

[mb]

Hi Norberto,

If you have ssh/shell access to your FW, run below command to see which processes are using the highest amount of memory:

# top -o res

No need to upgrade to 1.8.2 for this. 1.8.2 basically does the same thing.

[naltalef]

Hi. Thanks for your answer.
With Sensei running since 1 h aprox. this is is te output of top.

last pid: 47471;  load averages:  0.26,  0.22,  0.21  up 82+01:19:58    14:13:28
40 processes:  1 running, 39 sleeping
CPU:  0.9% user,  0.0% nice,  0.3% system,  0.1% interrupt, 98.7% idle
Mem: 3568M Active, 6243M Inact, 12M Laundry, 1740M Wired, 1204M Buf, 4824M Free
Swap: 8192M Total, 118M Used, 8074M Free, 1% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
37978 elasticsea   90  52    0    16G  2891M uwait    4   6:55   0.00% java
70777 root         11  20  -20  2365M  1186M nanslp   2   1:35   0.10% eastpect
33012 root         11  20  -20  2217M  1043M nanslp   1   1:31   0.00% eastpect
54255 root          1  20  -20  1318M   199M nanslp   0   0:01   0.00% eastpect
47906 root          1  52  -20  1318M   198M wait     0   0:00   0.00% eastpect
81766 root         41  20  -20   186M   108M select   0   0:26   0.00% python3.7
98584 root          1  22    0   132M    25M accept   0  13:55   0.00% python3.7
80229 root          1  20    0    40M    16M accept   3   0:00   0.00% php-cgi
76840 root          1  20    0    40M    16M accept   5   0:00   0.00% php-cgi
20743 root          1  20    0    41M    16M accept   0   0:00   0.00% php-cgi
70273 root          1  20    0    40M    16M accept   7   0:00   0.00% php-cgi
27950 root          1  20    0    40M    16M accept   7   0:00   0.00% php-cgi

I'll give you an update tomorrow.

Regards
Norberto

[rrosson]

I have a quad core i5 w with 16 gigs of ram and am running current version of opnsense and am seeing this as well. The only difference I chose to use mongodb instead of elasticsearch. When I click on the link to submit to vendor nothing happens and the box disappears using my primary browser (safari).

[rrosson]

Below is an output from top. Just had to restart opensense a few minutes ago:

Code: [Select]

last pid: 18745;  load averages:  0.24,  0.17,  0.16                                         up 6+00:49:28  08:26:24
86 processes:  2 running, 84 sleeping
CPU:  0.5% user,  0.0% nice,  0.7% system,  0.5% interrupt, 98.3% idle
Mem: 1617M Active, 5402M Inact, 4398M Laundry, 2589M Wired, 1430M Buf, 1943M Free
Swap: 8192M Total, 2893M Used, 5298M Free, 35% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 6606 mongodb      38  52    0  8120M  5214M uwait    0 103:38   0.26% mongod
70052 root         13  20    0  8218M  2449M nanslp   2 240:45   0.86% suricata
98128 clamav        3  20    0  1416M  1318M select   1  89:15   0.00% clamd
41124 root         11  20  -20  2461M   328M nanslp   1   0:01   0.29% eastpect
42907 root         11  20  -20  2459M   322M nanslp   2   0:01   0.38% eastpect
77240 root          1  20  -20  1269M   161M nanslp   0   0:00   0.00% eastpect
49873 root          1  52  -20  1269M   161M wait     0   0:00   0.00% eastpect
98455 squid         1  20    0  1491M   141M kqread   0  16:20   0.00% squid
64760 unbound       4  20    0   124M    95M kqread   2   8:46   0.00% unbound
52527 root         43  20  -20   125M    58M select   0   0:01   0.31% python3
87726 root          1  22    0    84M    43M accept   1   2:01   0.00% python3.7
 2686 root          1  52    0   190M    39M accept   2   0:01   0.00% php-cgi
18745 root          1  24    0    51M    39M CPU0     0   0:00   0.00% php
69463 root          1  22    0   188M    36M accept   2   0:03   0.00% php-cgi
87774 root          1  26    0   188M    36M accept   3   0:02   0.00% php-cgi
86102 root          1  20    0   186M    34M accept   2   0:00   0.00% php-cgi
 6591 root          1  20    0   188M    33M accept   3   0:02   0.00% php-cgi
22117 root          1  52    0   189M    33M accept   1   0:02   0.00% php-cgi



[sy]

Hi,

MongoDB has high usage. How many clients do you have behind the Sensei?