High memory usage

[dcol]

Finally fixed my SMTP for Monit and then started getting reports about high memory usage.

Two questions
1. How can I see what service is using the memory
2. Using 35% of 16GB. Is that normal? Don't remember it ever being that high.

I only use the basic features and Suricata. No VPN's, Proxy, or squid

[dcol]

Now my memory usage has creeped up to 60% of 16GB

What could be causing this?
How do I check which service is taking up memory?

Attached is a pic of my running services, if that helps.
Using version 19.7.4_1

[dcol]

Also got an error when doing a health audit.

[dcol]

Then there are also errors in the security audit as seen below.
What do I do at this point?

[franco]

I heard free memory is wasted, the system will hold on to free memory when it was acquired but give it to processes when it is needed (top output would help here).

One of your base files was rewritten, likely by a cron job in FreeBSD. The file is only for "man" console usage. You can reinstall the package "base" but the error could be back eventually when the cron job runs again.

For security errors... wait for update and/or read the references and protect your system accordingly in the meantime.


Cheers,
Franco

[dcol]

I do have 2 cron jobs. One is to update the ids rules and the other to update the alias.
These were setup over 2 years ago and has migrated through the updates.

Would either of these jobs cause the "base" package issue?

As far as the security errors, are these to be corrected in a future update, or is there something I need to do now?

I reinstalled "base" and the errors still appear

Also, looking at the details of these errors explain why I am getting such high memory usage now.
Looks like the fix is for OPNsense to update the vulnerable packages

[dcol]

Ok, this is now getting serious. I am up to 77% of 16GB memory usage.

I need some help. How do I tell what is using this memory? Never saw this before I updated to latest version.

Please......

[banym]

Have you tried to restart the services you use one by one to identify if memory is released after service restart?
Or go to the shell and try to identify the service or process with top or htop.
I would try to identify the process before restart any service to be more sure where the problem is located.

Are you using tmpfs btw?

[dcol]

The only service that made any difference was Suricata. but this morning I was only @35% and restarting Suricata only reduced it to 20% which is normal. I will have to do that again when it gets to over 70%.
So it does seem to release memory and drop down to into the 30's
Not sure what causes those memory usage spikes.

I do not have a tmpfs service installed.

[dcol]

This is getting serious. Had to restart OPNsense because it Suricata crashed.
I believe this is caused by suricata. Which has exceeded the memory to 100%.

This issue just started with the last update I did which was 19.7.2 to 19.7.4_1

I haven't changed the Suricata rules in ages. I do not see any errors in the Suricata logs but there is on concern in the backend

configd.py: encode idna: unable to decode DZ AO BJ BW BF BI CM CF TD CD CG CI DJ EG GQ ER ET GA GM GH GN GW KE LS LR LY MW ML MR MA MZ NA NE NG RW ST SN SL SO ZA SS SD SZ TZ TG TN UG EH ZM ZW AR BO BR CL CO SV GT HT HN NI PY SR VE AF AM AZ BH BD BT BN KH CN CY TL GE ID IR IQ JO KZ KP KW KG LA LB MO MY MN MM NP OM PK PS QA SA LK SY TJ TM AE UZ VN YE AL AD BY BA BG HR CZ HU LV LT RO RU RS SK SI TR UA JM PE HK IL IN JP KR PH SG TH TW CC CX IO KM MG MU MV RE SC TF YT AS CK FJ FM KI MH MP NC NF NR NU PF PG PN PW SB TK TO TV VU WF WS AX GG GI MD ME MK MT PL SM, return source

Could this be causing my issues?

Please help.........

[dcol]

Well, since I haven't paid much attention to rules and alias'es since version 18, I decided to clean up the alias'es.
memory usage has decreased and the previous reported error is gone.

I will see how it goes. If anyone has any other suggestions on cleaning up things, please share.

[dcol]

Back up to 75%+ memory usage.

Is this normal? Never seen this before update.

I'm afraid OPNsense will crash again like it did yesterday

[dcol]

This issue looks like a suricata problem. OPNsense crashed again and it appeared that all the Suricata rules were gone. They came back after a restart and the mem usage went back down to under 10%.

[franco]

Are you using Hyperscan? Try not using it...


Cheers,
Franco

[dcol]

I will try turning Hyperscan off if it happens again. I am now using 10-15% memory, which is what it normally is at. I did notice one of my newer local subnets was not included in the HOME_NETWORK list. And I was getting a DDOS attack from Germany which was being stopped by Suricata.

I think Suricata was flooded with hits and just blew up the firewall. I do notice that if I make any major changes to or restart the Suricata service. I no long have Internet and need to restart. But I assume this is normal.