After updating Opnsense from 25.7.10 to 25.7.11_1 opening WebGUI eats all memory

[wide]

I've updated from 25.7.10 to 25.7.11_1 yesterday. It's now impossible to access WebGUI because after the login screen when dashboard starts loading the UI spawns hundreds of PHP processes which causes system load to raise above 100 and after few minutes sytem runs out of memory and also consumes all the allocated swap space finally killing the network traffic completely.

System recovers from the situation after closing the WebGUI browser tab but it might take 30-60 minutes when all the PHP processes are finished and memory consumption and system load returns back to normal values.

[Patrick M. Hausen]

Spin up the system isolated and only your desktop/laptop connected to LAN. No WAN, no switch, no other devices.

System working? No problems? Also no Internet, of course.

If yes, go to Interfaces: Neighbors: Automatic Discovery and disable that. Then reconnect.

HTH,
Patrick

[wide]

Hi,

System is fully functional and stable after reboot if I don't open the WebGUI. So clean start and staying away from management keeps the load and memory consumption in similar levels what were before the 25.7.11_1 update. Is it sure that my case is connected to this Neighbors: Automatic Discovery feature?

[stanthewizzard]

Wrong post

[Patrick M. Hausen]

Is it sure that my case is connected to this Neighbors: Automatic Discovery feature?

No but this is the single change in the latest release which seems to impact folks negatively the most. Including filling disks, 100% CPU load, ... so I would not be surprised if it would also spawn a lot of processes.

[nuke]

I had exactly the same issue.  The disk filled up completely and all processes stopped.

It looks to be the logs in /var/log/hostwatch are so large that the disk is full and nothing else will work.  The GUI dies.

I had to reboot, connect locally to the OPNsense device, turn off/disable the Interfaces:Neighborhood (see post in German forum) and then manually delete the logs from /var/log/hostwatch.

Once the logs were deleted and I rebooted again, everything appears to be back to normal.

BTW, check out the other posts about hostwatch in the forum.  It appears they are all related to this.

[wide]

I managed to update to version 25.7.11_2 by using opnsense-shell and then run restart all the services from shell also.
System remains stable. No exessive disk writes, normal memory consuption and regular CPU load.

But still immediately after I open the WebGUI the systems goes haywire. Tens and then hundreds of PHP processes spawn and system runs out of memory.

[wide]

I was able to isolate this issue to WebGUI Dashboard. When I have shell open at the same time when I login to WebGUI and tens of php processes starts to spawn I then run killall php from the shell and then go to some other part of the WebGUI without any issues. So there is something in the Dashboard itself or in my particular Dashboard view which gets Opnsense to go haywire.

[wide]

Most likely this was not related to 25.7.11_1 update but changes in IDS configuration I had done right after the update. Finally today I restored the Opnsense VM from the backup taken before 25.7.11_1 update and updated the VM then to latest 25.7.11_2 and now all good so far.

[TCMSLP]

Signed up to the forum to report exactly the same issue.  The box appears stable until I login, then memory consumption quickly climbs to 100% and the UI becomes unresponsive.  I've tried disabling host / neighbourhood discovery but this makes zero difference.

Update 1:
I've now identified the problem process:
USER      PID  %CPU %MEM      VSZ     RSS TT  STAT STARTED      TIME COMMAND
root    51727   0.1 58.8 11386908 2316112  -  Ss   17:35     1:52.31 /usr/local/bin/suricata -D --netmap --pidfile /var/run/suricata.pid -c /usr/local/etc/suricata/suricata.yaml

After identifying this, I disabled intrusion detection and now everything is back to normal.   

Update 2:
Re-enabling IDS (and IPS) immediately causes the issue again.  However, I'm now wondering if new rules/changes may have increased memory usage; perhaps using the web UI is adding to this demand/exhaustion.  Either way, disabling IDS has solved my immediate problem.

OPNsense 25.7.11_2, 4GB RAM, i5-4570.

[franco]

Two people ending up identifying Suricata as the culprit on 25.7.11 when it was updated to 8.0.3? Hmm... try reverting, restarting the service and see if that's better.

# opnsense-revert -r 25.7.10 suricata


Cheers,
Franco