Hi everyone,
please let ma ask a question about the RAM usage.
AFter some time the RAM Usage increases to the max available ram. But i do not know why.
After looking at "System -> Diagnostics -> Activity" the are many Processes with 60M Ram size. It is called " /usr/local/bin/php /usr/local/etc/rc.newwanipv6 ppoe1 force (<php>)"
After rebooting the system, everything is working fine.
Someone got a similar problem ?
Thanks a lot
Mario
When you use ZFS as Filesystem it caches some files into the RAM. (ARC Cache)
If the file system is ufs maybe there are processes spawned that don't terminate completely (like the pppoe one you mentioned). If that's the case maybe it needs further investigation.
Yes, ZFS is used by this system.
> Is there any possibility to clear the cache after some time?
I wouldn't mess with the caching, the developers of ZFS were probably smarter than us. :)
Ok, thanks.
Some additonal info. A reboot is not working at the moment. Memory Usage still says 99% ram and 99% swap usage.
How much RAM does your system have?
The swap file being heavily used seems odd to me.
It got 8 GB and the Swap size is also 8 GB.
But there are not many Plugins installed. I've only installed Monit
HI,
at the moment the Swap usage is freed up. (at this time only 87%)
The only thing i changed is, to disable all IPv6 options on the Interfaces.
Maybe you found a process that spawns a huge number of child processes that don't terminate when thry're finished. You could check it with:
ps -axH
I cant help here anymore though since I don't know that much regarding system processes.
Hey,
i hope the atteched file will help you.
Thanks
Yeah I think it looks like a php script that doesn't terminate after it's done. (You were actually spot on in your first post :) )
52650 root 20 0 60M 5M CPU3 3 0:00 0.12% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force
This here spawns child processes till the RAM and SWAP is full.
Maybe a developer will read this and answer.
If you can replicate it easily you could also open an issue here with your full setup and what you're doing to get this issue:
https://github.com/opnsense/core/issues
Quote from: Monviech on October 13, 2023, 08:04:51 AM
If you can replicate it easily you could also open an issue here with your full setup and what you're doing to get this issue:
https://github.com/opnsense/core/issues
Thanks for your support. But I do not know when the issue appears. Sometimes it works fine for many days. And sometimes not.
But i will check if the failure also occurs when IPv6 is disabled.
At the moment the memory usage is 980/8071 MB of RAM and the ARC size is 260MB
Mario
I wouldn't worry too much unless processes start being killed.
The allocation of most of the memory is normal in freebsd with zfs, and is always good to have some swap.
I imagine it you use top or htop to have a live look at what is happening, you'd see the active amount of memory is low, even some swap has been used and eventually goes down.
You can try turning off all of your plugins and keeping an eye on the memory usage. That will help you tell if it's a core issue or not. Then slowly add them back one at a time and see what happens.
Another thing to watch is what your users are doing on the network. I saw a place chasing their tails trying to figure out why a server kept randomly going OOM until the realized it happened when a user attempted to upload a large file for processing. It could be that you're having something similar happen with torrents, game updates, etc.
True but to remain on the OP and avoid chasing ghosts unnecessarily: "AFter some time the RAM Usage increases to the max available ram. But i do not know why."
This is freebsd on zfs. Usage of most of RAM is normal.
Sorry to say but didn't you check the attached file the OP posted? You can see the broken child processes spawned in it. It's a php script that doesn't terminate.
theres like 200 instances that use 60mb each which fills the RAM and SWAP completely.
It has nothing to do with ZFS.
It doesn't appear for me. I am using a work machine working from home where all sorts of security stuff is applied that can prevent showing attachments and other potential sources of trouble.
That said that, the case might be of broken system but the principle is correct.
I'll post some of the content to make it clearer:
52650 root 20 0 60M 5M CPU3 3 0:00 0.12% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force
3897 root 24 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
97285 root 21 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
53475 root 23 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
53950 root 21 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
45364 root 23 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
86365 root 21 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
85887 root 21 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
39562 root 21 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
9674 root 36 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
7995 root 22 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
12936 root 21 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
73587 root 20 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
52016 root 23 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
66701 root 20 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
89709 root 22 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
24639 root 21 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
16225 root 20 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
5088 root 20 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
94137 root 20 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
42505 root 21 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
15714 root 21 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
12117 root 20 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
67147 root 21 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
95943 root 27 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
64509 root 20 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
2040 root 34 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
45929 root 20 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
34823 root 52 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
27995 root 38 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
8260 root 29 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
4159 root 29 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
64981 root 20 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
91548 root 32 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
60430 root 28 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
40812 root 52 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
46206 root 40 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
90404 root 37 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
24439 root 26 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
5051 root 40 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
22338 root 38 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
51501 root 22 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
79211 root 52 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
94557 root 52 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
38136 root 27 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
22450 root 29 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
48071 root 52 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
38410 root 20 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
13277 root 52 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
63490 root 21 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
86310 root 52 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
62889 root 21 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
77734 root 29 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
21835 root 20 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
94022 root 22 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
53991 root 31 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
4494 root 20 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
92529 root 31 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
23455 root 34 0 60M 8K lockf 3 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
7337 root 27 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
75932 root 52 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
70763 root 52 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
66199 root 27 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
25326 root 52 0 60M 8K lockf 0 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
86096 root 26 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
35722 root 27 0 60M 8K lockf 2 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
84149 root 27 0 60M 8K lockf 1 0:00 0.00% /usr/local/bin/php /usr/local/etc/rc.newwanipv6 pppoe1 force (<php>)
Ok thanks. That is something defo wrong that will just use up resources.
I'll also point out that zfs doesn't tend to chew up your swap. :)
Of course it doesn't but if you have processes using up memory, it is well known that ARC might not be freed fast enough before contents overspill to swap.
I've seen the attachment now and I although the zfs usage post I made is technically correct, i was off the mark with this one. These are definitively runaway processes.