Hi,
I have a strange issue where memory fill up very quickly and then swap get running out of space error causing the system/router crashed. This has been happening since last year and I have to reboot every 2 days before my whole network down. I tried to reduce the RAM from 16G to 4G (I thought it was required more RAM but the issue still happening but it delays the system crashed up to 1 weeks) and add 2G to swap (enabled in settings). I also reduce the plugins and the issue is still there. I also tried to deploy from scratch and import the config but the issue still persist. I attached my system log and I don't have any clue what was causing this, So I need your advice/help on this. Thanks in advance.
Regards,
Afif
Sounds like you're already aware of "System: Settings: Miscellaneous" -> "Disk / Memory Settings (reboot to apply changes)". Under normal circumstances I would recommend unchecking all three checkboxes, unless you have very weak non-volatile storage and can mitigate the issues. This is a guess, but worth a try.
Yes, let the defaults be. You don't need swap.
Seems like a virtual machine but that's not in itself a problem. Many run it on VM without problem.
If external monitoring is not setup or available, you need to regularly check for running processes to see which ones are using up the memory.
Telling what the setup and running services and plugins would help too.
Quote from: sopex on June 12, 2026, 11:34:16 PMYes, let the defaults be. You don't need swap.
IMHO running any system without swap is not a very good idea. If the system needs it for any reason, it is best for it to have it than not.
Quote from: cookiemonster on June 12, 2026, 11:39:59 PMIMHO running any system without swap is not a very good idea. If the system needs it for any reason, it is best for it to have it than not.
It's not a perfect science. In my limited experimentation, the chances of swap getting used and the whole system becoming sluggish are much higher than a 16GB+ system being starved of memory.
If creating swap is at all possible, it is hardly a bad idea to provide it to a system. Even when there is what appears to be plenty of memory for it.
It's quite easy to see most unix-like/*bsd systems with suitable amount of memory, they swap from time to time, even if in small amounts. For the sake of a few gigabytes of storage, it's an easy and cheap help to give it.
I already unchecked all the settings under disk/memory and restarted the router. FYI, this OPNsense is running inside the VM for 2+ years, so this issue is happening out of nowhere, and I can't identify which processes causing it to crash. I will let it run for 1-2 days and see if the router crash or not. I will also provide the info you are asking below.
VM SpecsCPUs: 4 cores
RAM: 4GB
Storage: 32GB
Running Services:- caddy
- configd
- cron
- ddclient
- dnsmasq
- hostwatch
- login
- ntpd
- openssh
- pf
- qemu-ga
- routing
- sysctl
- syslog-ng
- webgui
- wireguard
Plugins:- os-caddy
- os-ddclient
- os-q-feeds-connector
- os-qemu-guest-agent
Quote from: Afif on June 12, 2026, 06:19:17 PMThis has been happening since last year
Did you start using
Quote from: Afif on June 13, 2026, 07:20:40 AMRunning Services:
At the same moment perhaps ?
There have been a lot of strange issues with it since it was introduced !!
Quote from: nero355 on June 13, 2026, 08:20:16 PMQuote from: Afif on June 12, 2026, 06:19:17 PMThis has been happening since last year
Did you start using
Quote from: Afif on June 13, 2026, 07:20:40 AMRunning Services:
At the same moment perhaps ?
There have been a lot of strange issues with it since it was introduced !!
Tbh, I don't know about it. I thought that if every service running, it should be fine. Since you mentioned hostwatch service have a lot of strange issue, should I disable it?
Also, a bit update from my end. My RAM usage looks stable (within 25-35%), but it starts eating disk space little by little. Previously, it was 3.6GB and now it's already 4.6GB from the total of 32GB (12+ hours since I disabled the Temp RAM and SWAP settings). Should I be concerned about this?
Quote from: Afif on June 13, 2026, 08:37:33 PMTbh, I don't know about it. I thought that if every service running, it should be fine.
Since you mentioned hostwatch service have a lot of strange issue, should I disable it?
Considering the fact that you also wrote this :
QuoteAlso, a bit update from my end. My RAM usage looks stable (within 25-35%), but it starts eating disk space little by little. Previously, it was 3.6GB and now it's already 4.6GB from the total of 32GB (12+ hours since I disabled the Temp RAM and SWAP settings).
Should I be concerned about this?
YES! Absolutely!
FreeBSD as a "traditional" Unix does not have an explicit OOM killer. But in situations of memory pressure, whenever a process demands swap space while swap space is exhausted, it's killed.
However, if its eating up disk space and not RAM, perhaps its worth to have a look what fills the disk. The first suspect I would look at is the /var/log directory for the log files. Often enough a sick process throws a lot of log entries which fills disk space very fast
opnsense # du /var/log
might give some insights. Moreover, if its filling up a particular log file, this file might then give hints which process to look at
Br br
Quote from: nero355 on June 14, 2026, 12:24:56 AMYES! Absolutely!
Thanks Nero, I already disabled it.
Quote from: bringha on June 14, 2026, 09:32:08 AMHowever, if its eating up disk space and not RAM, perhaps its worth to have a look what fills the disk. The first suspect I would look at is the /var/log directory for the log files. Often enough a sick process throws a lot of log entries which fills disk space very fast
opnsense # du /var/log
might give some insights. Moreover, if its filling up a particular log file, this file might then give hints which process to look at
Br br
I attached the picture after running that command. Currently, its already 5.8GB used of disk space at the time I posted.
Quote from: Afif on June 14, 2026, 11:03:29 AM[...]Currently, its already 5.8GB used of disk space at the time I posted.
Are you using file system defaults (ZFS, compression)? The compression works very well on log files. (To check, "df -hT" and if ZFS, "zpool get feature@lz4_compress,feature@zstd_compress".)
Quote from: pfry on June 14, 2026, 02:31:40 PMAre you using file system defaults (ZFS, compression)? The compression works very well on log files. (To check, "df -hT" and if ZFS, "zpool get feature@lz4_compress,feature@zstd_compress".)
I'm not using ZFS on this system. Here is the output from the command.
root@OPNsense:/home/afif # df -hT
Filesystem Type Size Used Avail Capacity Mounted on
/dev/da0p2 ufs 29G 6.4G 20G 24% /
devfs devfs 1.0K 0B 1.0K 0% /dev
/dev/da0p1 msdosfs 256M 1.3M 255M 0% /boot/efi
root@OPNsense:/home/afif #
Quote from: Afif on June 14, 2026, 04:14:25 PMI'm not using ZFS on this system.[...]
Here's mine as an example. ~200 log files with an average size of ~90MB (file system defaults, firewall only, no netflow or hostwatch, most rule logging enabled, averaging a few hundred connections at any given moment), consuming... <3.3GB (I didn't bother to get a precise compression ratio).
root@fw:/home/user # df -hT
Filesystem Type Size Used Avail Capacity Mounted on
zroot/ROOT/default zfs 1.4T 1.4G 1.4T 0% /
devfs devfs 1.0K 0B 1.0K 0% /dev
/dev/gpt/efiboot0 msdosfs 260M 1.3M 259M 1% /boot/efi
zroot/var/mail zfs 1.4T 112K 1.4T 0% /var/mail
zroot zfs 1.4T 96K 1.4T 0% /zroot
zroot/tmp zfs 1.4T 3.4M 1.4T 0% /tmp
zroot/usr/ports zfs 1.4T 96K 1.4T 0% /usr/ports
zroot/home zfs 1.4T 140K 1.4T 0% /home
zroot/var/audit zfs 1.4T 96K 1.4T 0% /var/audit
zroot/var/tmp zfs 1.4T 96K 1.4T 0% /var/tmp
zroot/var/crash zfs 1.4T 96K 1.4T 0% /var/crash
zroot/var/log zfs 1.4T 3.3G 1.4T 0% /var/log
zroot/usr/src zfs 1.4T 96K 1.4T 0% /usr/src
devfs devfs 1.0K 0B 1.0K 0% /var/dhcpd/dev
root@fw:/home/user # zpool get feature@lz4_compress,feature@zstd_compress
NAME PROPERTY VALUE SOURCE
zroot feature@lz4_compress active local
zroot feature@zstd_compress enabled local
root@fw:/home/user # ls -la /var/log/filter
total 3252314
drwx------ 2 root wheel 203 Jun 14 08:01 .
drwxr-xr-x 17 root wheel 39 Jun 14 03:01 ..
-rw------- 1 root wheel 87636510 Nov 27 2025 filter_20251127.log
-rw------- 1 root wheel 113842291 Nov 28 2025 filter_20251128.log
-rw------- 1 root wheel 66937381 Nov 29 2025 filter_20251129.log
-rw------- 1 root wheel 63962720 Nov 30 2025 filter_20251130.log
-rw------- 1 root wheel 65829378 Dec 1 2025 filter_20251201.log
-rw------- 1 root wheel 59705785 Dec 2 2025 filter_20251202.log
-rw------- 1 root wheel 65167863 Dec 3 2025 filter_20251203.log
-rw------- 1 root wheel 58694081 Dec 4 2025 filter_20251204.log
-rw------- 1 root wheel 57220899 Dec 5 2025 filter_20251205.log
-rw------- 1 root wheel 58380288 Dec 6 2025 filter_20251206.log
[...]
-rw------- 1 root wheel 112167589 Jun 5 23:59 filter_20260605.log
-rw------- 1 root wheel 100272143 Jun 6 23:59 filter_20260606.log
-rw------- 1 root wheel 89032171 Jun 7 23:59 filter_20260607.log
-rw------- 1 root wheel 97640115 Jun 8 23:59 filter_20260608.log
-rw------- 1 root wheel 96846494 Jun 9 23:59 filter_20260609.log
-rw------- 1 root wheel 85642933 Jun 10 23:59 filter_20260610.log
-rw------- 1 root wheel 88817261 Jun 11 23:59 filter_20260611.log
-rw------- 1 root wheel 91362771 Jun 12 23:59 filter_20260612.log
-rw------- 1 root wheel 89815402 Jun 13 23:59 filter_20260613.log
-rw------- 1 root wheel 35648453 Jun 14 08:57 filter_20260614.log
lrwxr-x--- 1 root wheel 35 Jun 14 08:01 latest.log -> /var/log/filter/filter_20260614.log
root@fw:/home/user #