Text only | Text with Images

OPNsense Forum

English Forums => 25.1, 25.4 Production Series => Topic started by: Bights on March 04, 2025, 11:49:03 AM

Title: Help! After updating, memory keeps filling and crashing system
Post by: Bights on March 04, 2025, 11:49:03 AM
Help!


Since updating yesterday to OPNsense 25.1.2-amd64 with all patches, the memory keeps getting consumed to full resulting in full instability.  Only way to recover is a reboot, but the cycle starts again within minutes.

Seems likely this may be due to lighttpd constantly starting and stopping, as seen in the WebGUI logs:


2025-03-04T10:20:53   Notice   lighttpd   (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.77/src/server.c.1943) server started (lighttpd/1.4.77)   
2025-03-04T10:20:53   Notice   lighttpd   (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.77/src/server.c.2339) server stopped by UID = 0 PID = 72132   
2025-03-04T10:20:53   Notice   lighttpd   (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.77/src/server.c.1234) [note] graceful shutdown started   
2025-03-04T10:19:33   Notice   lighttpd   (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.77/src/server.c.1943) server started (lighttpd/1.4.77)   
2025-03-04T10:18:43   Notice   lighttpd   (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.77/src/server.c.1234) [note] graceful shutdown started   
2025-03-04T10:18:39   Notice   lighttpd   (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.77/src/server.c.1943) server started (lighttpd/1.4.77)   
2025-03-04T10:18:39   Notice   lighttpd   (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.77/src/server.c.2339) server stopped by UID = 0 PID = 48479   
2025-03-04T10:18:36   Notice   lighttpd   (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.77/src/server.c.1234) [note] graceful shutdown started
...


Also there seems to be an unusually high count (350+) processes matching this line:
   259   root   68   0   229M   135M   wait   2   0:00   0.00%   /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.11){python3.11}


...and a similar number (350+) of processes matching this line:
   66432   root   68   0   58M   32M   lockf   1   0:00   0.00%   /usr/local/bin/php /usr/local/etc/rc.newwanipv6 igc0 force


The process count increases all the time, now it's closer to 400, with memory utilisation at 90% (I have 8Gb)


Help!!


Thanks!


Title: Re: Help! After updating, memory keeps filling and crashing system
Post by: Bights on March 04, 2025, 11:49:46 AM
Load average
3.07, 3.57, 2.36
Title: Re: Help! After updating, memory keeps filling and crashing system
Post by: Bights on March 04, 2025, 11:52:25 AM
Process count of python now 472 and climbing...  Memory utilisation at 98%.   Rebooting again.... :(
Title: Re: Help! After updating, memory keeps filling and crashing system
Post by: Patrick M. Hausen on March 04, 2025, 11:53:59 AM
As a first attempt at mitigation I suggest disabling IPv6 on WAN.
Title: Re: Help! After updating, memory keeps filling and crashing system
Post by: Bights on March 04, 2025, 11:59:54 AM
Thanks will do...
Title: Re: Help! After updating, memory keeps filling and crashing system
Post by: Bights on March 04, 2025, 12:05:01 PM
Whether it was turning off DHCPv6 or the most recent reboot, the load is now 0.26, 0.27, 0.17 and memory at ~21% and holding.

"configd.py" processes are at 3, "rc.newwanipv6" at 0.

Will continue to monitor.

Thanks again.
Title: Re: Help! After updating, memory keeps filling and crashing system
Post by: Patrick M. Hausen on March 04, 2025, 12:19:55 PM
Do you need IPv6 and does your ISP provide it? If yes, of course this should be investigated further.
Title: Re: Help! After updating, memory keeps filling and crashing system
Post by: Bights on March 04, 2025, 12:36:21 PM
My ISP uses IPv4, so thankfully this issue for me at least appears to be resolved by disabling DHCPv6 (memory utilisation stable even now).

Title: Re: Help! After updating, memory keeps filling and crashing system
Post by: jim1985 on March 04, 2025, 02:55:36 PM
This was exactly the same problem I have been having & discovered yesterday that disabling IPv6 on the WAN was the solution.


My ISP also only provides IPv4



I wonder, is there a way that OPNsense could be told to disable IPv6 automatically if it detects that it is not available?
Text only | Text with Images