Swap used to 70%?

PotatoCarl · September 10, 2020, 11:36:00 AM

Hi
I use an appliance from Deciso with my OPNSense. I am wondering why I get a swap use of about 70% (5GB).
I am pretty sure that it is not a very good idea to Swap on the SSD all the time, but I do not find a good overview, which program is actually swaping. Maybe I just overlooked it but would apreaciate a hint.
Cheers

siga75 · September 10, 2020, 01:19:52 PM

you don't want to know the program that has memory swapped out, you should find the process that eat all the RAM and makes other processes to swap out

top command gives you an overview of the memory used by processes, except shared segments, for which you can use "ipcs -a"

to have information on the process being swapped out

[root@myfw ~]# swapinfo
Device 1K-blocks Used Avail Capacity
/dev/gpt/swapfs 8388608 408476 7980132 5%
[root@myfw ~]# ps auxvw
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND SL RE PAGEIN LIM TSIZ
squid 7496 0.0 1.3 1588020 213728 - S 3Sep20 58:40.26 (squid-1) --kid 0 127 3042 - 4472
squid 30228 0.0 0.0 1249468 648 - Is 3Sep20 0:00.00 /usr/local/sbin/ 127 127 27 - 4472
root 16852 0.1 5.9 3443616 986580 - Ss Mon16 50:24.28 /usr/local/bin/s 0 127 2592 - 3904
www 57923 0.0 0.0 197984 3692 - Ss 3Sep20 0:34.39 /usr/local/sbin/ 0 127 26 - 1992
root 4326 0.0 0.0 39632 56 - I 3Sep20 0:00.00 php-fpm: pool we 127 127 0 - 2532
www 14485 0.0 0.0 39632 56 - I 3Sep20 0:00.00 php-fpm: pool ww 127 127 0 - 2532
root 44326 0.0 0.0 39632 56 - I 3Sep20 0:00.00 php-fpm: pool we 127 127 0 - 2532
www 60404 0.0 0.0 39632 56 - I 3Sep20 0:00.00 php-fpm: pool ww 127 127 0 - 2532
root 97642 0.0 0.0 39556 3904 - Ss 3Sep20 0:22.54 php-fpm: master 0 127 0 - 2532
root 12647 0.0 0.5 104164 78140 - I Sun19 0:20.92 /usr/local/bin/p 127 127 11 - 2468
root 19873 0.0 0.3 82408 56468 - I Sun19 0:01.67 /usr/local/bin/p 127 127 0 - 2468
root 35072 0.0 0.4 90584 63668 - I Sun15 0:21.20 /usr/local/bin/p 127 127 78 - 2468
root 57400 0.0 0.0 39428 424 - Is 3Sep20 0:00.03 /usr/local/bin/p 127 127 74 - 2468
root 63011 0.0 0.0 39428 440 - Is 3Sep20 0:00.08 /usr/local/bin/p 127 127 28 - 2468
root 63827 0.0 0.3 75116 48852 - I Sun16 0:12.66 /usr/local/bin/p 127 127 0 - 2468
root 67420 0.0 0.2 59172 34284 - I Sun19 0:19.97 /usr/local/bin/p 127 127 0 - 2468
root 88293 0.0 0.4 89840 64328 - I Sun19 0:21.45 /usr/local/bin/p 127 127 3 - 2468
root 22284 0.0 0.0 13860 996 - Ss 3Sep20 0:00.39 /sbin/devd 5 127 19 - 720
root 1 0.0 0.0 9588 152 - ILs 3Sep20 0:00.54 /sbin/init -- 52 127 31 - 648
dhcpd 88018 0.0 0.0 23408 2732 - Ss 3Sep20 0:17.88 /usr/local/sbin/ 0 127 62 - 2344
root 20077 0.0 0.0 1073960 3368 - Ss 3Sep20 0:26.05 /usr/local/sbin/ 0 127 19 - 636
root 52944 0.0 0.0 1070056 5004 0 I 13:02 0:00.01 /bin/csh 127 127 20 - 632
root 71275 0.0 0.0 1069752 5200 0 S 13:02 0:00.02 bash 0 127 5 - 660
root 43759 0.0 0.0 1071804 3188 - Is 3Sep20 0:00.00 sshd: /usr/local 127 127 14 - 604
root 76191 0.0 0.0 1072348 7456 - Ss 13:02 0:00.06 sshd: root@pts/0 0 127 0 - 604
www 9222 0.0 0.0 362960 1272 - S 3Sep20 0:04.10 nginx: cache man 9 127 4 - 780
root 23327 0.0 0.0 1065424 4452 - Ss 3Sep20 0:05.72 /usr/local/sbin/ 6 127 24 - 524
root 45344 0.0 0.0 362924 1000 - Is 3Sep20 0:02.88 nginx: master pr 52 127 0 - 780
root 61518 0.0 0.0 1071808 3292 - Ss 3Sep20 0:04.76 /usr/local/sbin/ 9 127 5 - 524
www 90659 0.0 0.0 360732 7136 - S 3Sep20 2:21.98 nginx: worker pr 18 127 226 - 780
redis 84171 0.0 0.0 15356 1620 - Ss 3Sep20 2:56.39 redis-server: /u 0 127 11 - 828
unbound 35027 0.0 0.2 74604 29580 - Ss 3Sep20 17:52.12 /usr/local/sbin/ 18 127 3159 - 720
root 36429 0.0 0.0 1061540 1300 - Ss 3Sep20 1:58.05 /bin/sh /var/db/ 8 127 0 - 188
root 14765 0.0 0.0 1061388 3240 0 Is 13:02 0:00.00 /bin/sh /usr/loc 127 127 0 - 188
squid 11764 0.0 0.0 1066996 3264 - S Sat00 0:11.69 (pinger) (pinger 0 127 0 - 92
squid 11860 0.0 0.0 1073320 7224 - S Mon00 0:07.66 (pinger) (pinger 0 127 29 - 92
squid 35898 0.0 0.0 1067160 3324 - S Sun00 0:09.81 (pinger) (pinger 0 127 4 - 92
squid 41761 0.0 0.0 1067140 7452 - S Mon00 0:07.96 (pinger) (pinger 0 127 0 - 92
squid 42796 0.0 0.0 1067184 3264 - S 3Sep20 0:13.89 (pinger) (pinger 0 127 4 - 92
squid 48766 0.0 0.1 1067128 8696 - S Tue00 0:06.01 (pinger) (pinger 0 127 0 - 92
squid 54168 0.0 0.1 1067128 8696 - S 00:00 0:01.41 (pinger) (pinger 0 127 0 - 92
squid 55264 0.0 0.1 1067164 8700 - S Wed00 0:03.71 (pinger) (pinger 0 127 0 - 92
squid 74367 0.0 0.0 1067184 3268 - S Fri00 0:13.59 (pinger) (pinger 0 127 0 - 92
squid 38597 0.0 0.0 1072044 6808 - I Mon00 0:01.81 (security_file_c 127 127 0 - 116
squid 90190 0.0 0.0 1065660 6496 - I Mon00 0:00.04 (security_file_c 127 127 2 - 116
root 62767 0.0 0.0 1060672 1360 - Is 3Sep20 0:00.00 /usr/sbin/moused 127 127 0 - 76
root 35181 0.0 0.0 1061384 3064 0 R+ 13:15 0:00.00 ps auxvw 127 0 0 - 72
root 31610 0.0 0.0 1061276 2020 - I 13:00 0:00.00 cron: running jo 127 127 0 - 76
root 44857 0.0 0.0 1061276 1068 - Is 3Sep20 0:10.65 /usr/sbin/cron - 47 127 0 - 76
root 95276 0.0 0.0 1060708 1368 u0 Is+ 3Sep20 0:00.00 /usr/libexec/get 127 127 1 - 64
root 20043 0.0 0.0 1060708 1372 v0 Is+ 3Sep20 0:00.00 /usr/libexec/get 127 127 2 - 64
root 29063 0.0 0.0 1060708 1368 v1 Is+ 3Sep20 0:00.00 /usr/libexec/get 127 127 0 - 64
root 99709 0.0 0.0 1060708 1368 v2 Is+ 3Sep20 0:00.00 /usr/libexec/get 127 127 0 - 64
root 11383 0.0 0.0 1060708 1368 v3 Is+ 3Sep20 0:00.00 /usr/libexec/get 127 127 0 - 64
root 66073 0.0 0.0 1060708 1368 v4 Is+ 3Sep20 0:00.00 /usr/libexec/get 127 127 0 - 64
root 33776 0.0 0.0 1060708 1372 v5 Is+ 3Sep20 0:00.00 /usr/libexec/get 127 127 0 - 64
root 88360 0.0 0.0 1060708 1368 v6 Is+ 3Sep20 0:00.00 /usr/libexec/get 127 127 0 - 64
root 19616 0.0 0.0 1060708 1368 v7 Is+ 3Sep20 0:00.00 /usr/libexec/get 127 127 0 - 64
nobody 23662 0.0 0.0 1060552 248 - S 3Sep20 0:07.21 /usr/local/bin/s 1 127 1 - 60
root 60553 0.0 0.0 1061692 1680 - Ss 3Sep20 0:22.95 /usr/local/sbin/ 0 127 1 - 60
root 80167 0.0 0.0 1060668 1388 - Is 3Sep20 0:00.00 daemon: /usr/loc 127 127 0 - 56
root 7913 0.0 0.0 1060504 2476 - SC 13:15 0:00.00 sleep 60 8 8 0 - 52
root 22429 0.0 0.0 15708 3276 - Ss 3Sep20 0:04.11 /usr/local/sbin/ 3 127 48 - 384
zabbix 8174 0.0 0.0 23836 3772 - S 3Sep20 0:39.14 zabbix_agentd: l 1 127 11 - 304
zabbix 69630 0.0 0.0 24056 3824 - S 3Sep20 0:58.02 zabbix_agentd: c 0 127 2 - 304
zabbix 81941 0.0 0.0 24116 3480 - S 3Sep20 0:36.61 zabbix_agentd: a 0 127 12 - 304
zabbix 85987 0.0 0.0 23900 1836 - S 3Sep20 0:38.47 zabbix_agentd: l 1 127 9 - 304
zabbix 99597 0.0 0.0 24056 0 - IW - 0:00.00 /usr/local/sbin/ 127 127 0 - 304
root 14694 0.0 0.0 15188 2692 - S 3Sep20 0:20.04 /usr/local/sbin/ 0 127 0 - 208
root 45792 0.0 0.0 17616 5172 - S 3Sep20 0:29.13 /usr/local/sbin/ 0 127 21 - 208
c_icap 21663 0.0 0.0 22808 2812 - S 00:00 0:01.57 /usr/local/bin/c 0 127 0 - 88
c_icap 28267 0.0 0.0 21644 2464 - Ss 3Sep20 0:20.35 /usr/local/bin/c 0 127 73 - 88
c_icap 46587 0.0 0.0 41260 3988 - S 00:00 0:01.94 /usr/local/bin/c 0 127 0 - 88
clamav 72579 0.0 6.8 1200404 1127384 - Is 3Sep20 18:23.29 /usr/local/sbin/ 127 127 27739 - 64
_flowd 44621 0.0 0.0 11452 1556 - Ss 3Sep20 0:16.02 flowd: net (flow 1 127 27 - 72
root 88564 0.0 0.0 10992 288 - Is 3Sep20 0:00.00 flowd: monitor ( 127 127 16 - 72
clamav 76927 0.0 0.0 27772 8284 - Is 3Sep20 0:09.15 /usr/local/bin/f 127 127 370 - 36
root 40706 0.0 0.0 13944 1488 - Is 3Sep20 0:59.92 /usr/local/bin/d 127 127 4 - 12
root 110 0.0 0.1 98228 19908 - I 3Sep20 2:02.45 /usr/local/bin/p 53 127 113 - 4
root 10197 0.0 0.0 33940 6488 - Ss 3Sep20 35:55.09 /usr/local/sbin/ 0 127 60 - 4
root 19612 0.0 0.2 42596 33048 - Is 13:00 0:00.45 /usr/local/bin/p 127 127 0 - 4
root 30558 0.0 0.1 31788 14568 - Ss 3Sep20 311:12.30 /usr/local/bin/p 0 127 61 - 4
root 37577 0.0 0.0 21472 6872 - S 3Sep20 0:58.74 /usr/local/bin/p 0 127 229 - 4
root 72298 0.0 0.0 18232 0 - IW - 0:00.00 /usr/local/sbin/ 127 127 0 - 4
root 85124 0.0 0.0 33004 0 - IWs - 0:00.00 /usr/local/bin/p 127 127 0 - 4
root 0 0.0 0.0 0 560 - DLs 3Sep20 121:22.01 [kernel] 9 127 0 - 0
root 2 0.0 0.0 0 16 - DL 3Sep20 0:00.00 [crypto] 127 127 0 - 0
root 3 0.0 0.0 0 16 - DL 3Sep20 0:00.00 [crypto returns 127 127 0 - 0
root 4 0.0 0.0 0 16 - DL 3Sep20 0:00.00 [crypto returns 127 127 0 - 0
root 5 0.0 0.0 0 16 - DL 3Sep20 0:00.00 [crypto returns 127 127 0 - 0
root 6 0.0 0.0 0 16 - DL 3Sep20 0:00.00 [crypto returns 127 127 0 - 0
root 7 0.0 0.0 0 32 - DL 3Sep20 0:00.00 [cam] 127 127 0 - 0
root 8 0.0 0.0 0 16 - DL 3Sep20 0:00.16 [soaiod1] 29 127 0 - 0
root 9 0.0 0.0 0 16 - DL 3Sep20 0:00.16 [soaiod2] 28 127 0 - 0
root 10 0.0 0.0 0 16 - DL 3Sep20 0:00.00 [audit] 127 127 0 - 0
root 11 400.0 0.0 0 64 - RNL 3Sep20 39488:27.95 [idle] 127 127 0 - 0
root 12 0.0 0.0 0 240 - WL 3Sep20 145:01.17 [intr] 127 127 0 - 0
root 13 0.0 0.0 0 48 - DL 3Sep20 0:00.01 [geom] 127 127 0 - 0
root 14 0.0 0.0 0 16 - DL 3Sep20 0:00.00 [sequencer 00] 127 127 0 - 0
root 15 0.0 0.0 0 80 - DL 3Sep20 0:25.17 [usb] 127 127 0 - 0
root 16 0.0 0.0 0 16 - DL 3Sep20 0:00.16 [soaiod3] 2 127 0 - 0
root 17 0.0 0.0 0 16 - DL 3Sep20 0:00.16 [soaiod4] 3 127 0 - 0
root 18 0.0 0.0 0 16 - DL 3Sep20 0:00.00 [sctp_iterator] 127 127 0 - 0
root 19 0.0 0.0 0 16 - DL 3Sep20 3:24.54 [pf purge] 0 127 0 - 0
root 20 0.0 0.0 0 16 - DL 3Sep20 5:33.70 [rand_harvestq] 0 127 0 - 0
root 21 0.0 0.0 0 16 - DL 3Sep20 0:07.08 [acpi_thermal] 7 127 0 - 0
root 3822 0.0 0.0 0 16 - DL 3Sep20 0:05.86 [vnlru] 0 127 0 - 0
root 20454 0.0 0.0 0 64 - DL 3Sep20 0:03.24 [ng_queue] 11 127 0 - 0
root 32263 0.0 0.0 0 144 - DL 3Sep20 1:35.75 [bufdaemon] 0 127 0 - 0
root 48050 0.0 0.0 0 48 - DL 3Sep20 2:27.99 [pagedaemon] 0 127 0 - 0
root 87973 0.0 0.0 0 16 - DL 3Sep20 0:01.01 [vmdaemon] 127 127 0 - 0
root 91598 0.0 0.0 0 16 - DL 3Sep20 6:20.25 [syncer] 0 127 0 - 0

Vilhonator · September 10, 2020, 02:13:11 PM

There also could be more simpler reason for huge swap usage, have you rebooted your opnsense device recently?

Most common reason why people face huge swap usage is because they don't reboot their systems and keep them up and running (RAM does not empty itself AT ALL as long as system is up and running.)

If your opnsense has around 4GB RAM on it, it is possible that because you haven't rebooted it for a while, your system has just ran out of RAM and now uses SWAP instead. Simple reboot should fix that.

I would highly recommend setting automated reboot (shown on the image), that way you don't have to remember anything other than take backups before or after reboot, and that internet doesn't work at 00:00 AM local time first day of every month (if you schedule it like I do, and still panic when I get disconnected in middle of a raid boss fight every month -_-)

siga75 · September 10, 2020, 03:59:16 PM

Quote from: Vilhonator on September 10, 2020, 02:13:11 PM
There also could be more simpler reason for huge swap usage, have you rebooted your opnsense device recently?

Most common reason why people face huge swap usage is because they don't reboot their systems and keep them up and running (RAM does not empty itself AT ALL as long as system is up and running.)

If your opnsense has around 4GB RAM on it, it is possible that because you haven't rebooted it for a while, your system has just ran out of RAM and now uses SWAP instead. Simple reboot should fix that.

I would highly recommend setting automated reboot (shown on the image), that way you don't have to remember anything other than take backups before or after reboot, and that internet doesn't work at 00:00 AM local time first day of every month (if you schedule it like I do, and still panic when I get disconnected in middle of a raid boss fight every month -_-)

This is not true at all! RAM is free when the processes release it or when they finish

If the system used the swap is because it had a lack of RAM at some point, then it swapped. After that, even if you have free RAM, the system does not put back the swap in it just because it is non sense. If a process will need to access that part of swapped out memory then a that point a swap in happens.

If you see RAM almost always at 100% or close it's just because, like every unix systems, freeBSD is smart and use all the needed and available memory for file system caching, the biggest part of the FS caching (the non dirty pages) can be released in an instant, if needed by a process.

Of course a reboot will free up the swap if you don't like to see swap usage, but this is not needed, it's just cosmetics

Vilhonator · September 10, 2020, 05:52:18 PM

Quote from: siga75 on September 10, 2020, 03:59:16 PM
This is not true at all! RAM is free when the processes release it or when they finish

Oh so service or process which has 4GBs reserved to RAM or SWAP file can release 8GBs of ram used by 20 different services or processes at that time? Yay

siga75 · September 10, 2020, 06:07:08 PM

Quote from: Vilhonator on September 10, 2020, 05:52:18 PM
Quote from: siga75 on September 10, 2020, 03:59:16 PM
This is not true at all! RAM is free when the processes release it or when they finish

Oh so service or process which has 4GBs reserved to RAM or SWAP file can release 8GBs of ram used by 20 different services or processes at that time? Yay

You just don't know what you are talking about. I will not even reply.
Good luck with your "Windows Style" periodic reboot

PotatoCarl · September 11, 2020, 09:04:31 AM

Hi
so when using "ps auxvw" this is little helpful, as the command line is cut so I cannot see the process.

So when I use

Code Select

ps auxwm I get a much clearer picture. Unfortunately, the "m" option does not, as it is supposed to do, sort by memory usage.

Going manually through it, "clamd" uses about 40.6% (!), maltrail with several processes 11.6 and 3x 12.2%

I use clamd with i-cap for squid, but I am amazed about the memory usage. As I am using on other servers clamd (among other virus scanners) for mail scanning, this seems to be a lot of memory to use.
I removed the plugin, however, it is interestingly still shown in the "ps" output.
The appliance only has 4GB ram, however, as it runs headless I am assuming that should be more than enough.
After a reboot the memory usage is more to normal "Swap=0".

Thanks.

Tubs · September 13, 2020, 03:37:51 AM

I have the same issue sind 20.7.x.

Before the system was running fine with 8 GB RAM installed and plenty of free RAM. No other change beside upgrade to 20.7.x. In my case after a while Sensei is running out of memory, stops automatically and is getting disabled. After Sensei stopped all is fine. But if Sensei is cause or effect I cannot say.

Code Select

Sensei has detected a problem during operation and has shut down Sensei services in order to prevent a network outage.

It is because we detected high SWAP usage 36 % ( 2.95GB / 8GB )

If you think this is something we should have a look, just click here to let us know about the details and we will investigate this further.

You can re-enable the services from Status page.

GreenMatter · November 24, 2020, 12:45:20 PM

Quote from: Tubs on September 13, 2020, 03:37:51 AM
I have the same issue sind 20.7.x.

Before the system was running fine with 8 GB RAM installed and plenty of free RAM. No other change beside upgrade to 20.7.x. In my case after a while Sensei is running out of memory, stops automatically and is getting disabled. After Sensei stopped all is fine. But if Sensei is cause or effect I cannot say.

Code Select Expand
Sensei has detected a problem during operation and has shut down Sensei services in order to prevent a network outage. It is because we detected high SWAP usage 36 % ( 2.95GB / 8GB ) If you think this is something we should have a look, just click here to let us know about the details and we will investigate this further. You can re-enable the services from Status page.

I keep getting same as above - SWAP being used over 30% and Sensei gets disabled. Prior to 20.7, SWAP almost never been used. And 8GB of RAM is utilized at about 60% or less as average maximum. Do I need to increase RAM or only change settings of "something"?

Tubs · November 29, 2020, 02:04:17 PM

Since 20.7.4 it is better again. Still more swap is used than before. But no service is stopping any more.

Swap used to 70%?

PotatoCarl

September 10, 2020, 11:36:00 AM

siga75

September 10, 2020, 01:19:52 PM #1

Vilhonator

September 10, 2020, 02:13:11 PM #2

siga75

September 10, 2020, 03:59:16 PM #3

Vilhonator

September 10, 2020, 05:52:18 PM #4

siga75

September 10, 2020, 06:07:08 PM #5

PotatoCarl

September 11, 2020, 09:04:31 AM #6

Tubs

September 13, 2020, 03:37:51 AM #7

GreenMatter

November 24, 2020, 12:45:20 PM #8

Tubs

November 29, 2020, 02:04:17 PM #9