OPNsense unresponsive after a week - Swap full

[chris42]

Hi there,
I have a weird phenomenon. OPNsense works fine for about a week, but then it completely becomes unresponsive. No routing, no webUI, nothing.
Have to log in via serial console. On there, I get only the following message repeated:

Code: [Select]

swp_pager_getswapspace(2): failed
swap_pager: out of swap space
I am not sure, if out of swap is the error causing this or a result of another error (e.g., then filling up the logs).
Not sure on how to analyze this properly. Any ideas?

Kind regards
  Chris

[cookiemonster]

Swap is normally a file or a partition of the disk, so only indirectly related to logs filling space.
You probably have  many services/activities that cause the system to consume all memory and then swap. At that point, the system will become unresponsive.
Verify you have disk space for completeness, then monitor a couple of times a day, say morning and afternoon (after a reboot) using usual top, htop, ps, etc. for what is consuming your main memory.
Also, are you using "Disk / Memory Settings" from System | Settings | Miscellaneous ?

[chris42]

This happened already a week ago, so was monitoring the week and checking, but RAM and swap was ok. This setup runs for some years now, so not sure what happened to it, that I now get that situation. I have minimal services running on the box, so doubt that it is overloaded.
I had log2ram activated, that is why I suspected an out of memory due to some service log spamming. However logs are not available after the "crash". I now deactivated log2ram, hence expect to see something now on disk. That might give better clues on what is happening.
Last things I changed is switching to native ddclient and testing some virtual ULA ipv6 addresses.

[chris42]

So after some observation, I now found, that I have massive logspam with some routing information:

Code: [Select]

<13>1 2024-03-01T23:05:33+01:00 opnsense.xxx.xxx kernel - - [meta sequenceId="6806"] <7>cannot forward src fe80:b::xxxx:xxxx:xxxx:xxxx, dst xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx, nxt 6, rcvif bridge0, outif pppoe1
I get that nearly every 10 seconds. Which is slowly filling my log ramdisk.
For some reason the DHCP6 crashed / is not running anymore.
Not sure if this is all connected, but something is wrong.