OPNsense Forum
Archive => 15.7 Legacy Series => Topic started by: BrianLloyd on November 27, 2015, 06:06:02 am
-
I have begun to have a problem with my WiFi clients not being able to get assigned their configuration with DHCP. It started a couple days ago with my iPhone randomly not being able to access the net because it does not have an assigned IP address. I rebooted last night and everything came back. This evening the problem is back. First thing I noticed was that the FS is full.
/ (ufs): 109% used 14G/14G
WTF? What would be eating that much disk? This is a pretty basic installation. So off to the logs. The system log is full of:
kernel: pid 73581 (dhcpd), uid 136 inumber 1364898 on /mnt: filesystem full
and
kernel: pid 59843 (suricata), uid 0 inumber 1364877 on /mnt: filesystem full
So it stands to reason that dhcpd has a problem not being able to write out the leases. That would answer why the clients aren't getting assigned addresses. But I'm still at a loss as to what would consume all the disk and cause the problem in the first place. how to clear it out again.
Oh, and I stopped intrusion detection just in case. It does seem to be trying to write a lot of stuff to disk.
Help?
-
We don't write /mnt as far as I know. Knowing Suricata it might have spammed /var/log, please provide the following output for further assistance. :)
# ls -lah /var/log
# ls -lah /var/log/squid
# ls -lah /var/log/suricata
-
Think we have a winner for mine:
ls -lah /var/log/suricata
total 16589428
drwx------ 2 root wheel 512B Nov 27 00:00 .
drwxr-xr-x 6 root wheel 1.5K Nov 27 00:00 ..
-rwx------ 1 root wheel 0B Nov 22 23:00 eve.json
-rwx------ 1 root wheel 0B Nov 22 23:00 eve.json.0
-rwx------ 1 root wheel 594B Nov 15 23:00 eve.json.1
-rwx------ 1 root wheel 16G Nov 8 23:00 eve.json.2
-rwx------ 1 root wheel 231K Nov 1 23:00 eve.json.3
-rwx------ 1 root wheel 8.8M Nov 27 08:16 stats.log
-rwx------ 1 root wheel 37M Nov 27 00:00 stats.log.0
-rwx------ 1 root wheel 32M Nov 26 00:00 stats.log.1
-rwx------ 1 root wheel 26M Nov 25 00:00 stats.log.2
-rwx------ 1 root wheel 37M Nov 24 00:00 stats.log.3
-rwx------ 1 root wheel 1.8M Nov 23 00:00 stats.log.4
-rwx------ 1 root wheel 43M Nov 22 00:00 stats.log.5
-rwx------ 1 root wheel 37M Nov 21 00:00 stats.log.6
Removed the offending file: eve.json.2
Gone down to:
/ (ufs): 4% used 1.1G/28G
-
Thanks. Well, the issue is moot for me at this point. Shortly after writing my original posting the web interface displayed an error message about there not being a config file instead of showing the relevant page. I (stupidly) rebooted the machine. OPNsense did not come back up and the LAN interface never responded to pings. I think I am going to have to rebuild the system. Fortunately for me I was able to fall back to my m0n0wall system and keep my network running.
Clearly there is some kind of problem with something filling up the disk storage. I suspect the right answer is to either run a utility to roll the logs and delete old ones when /var/log gets too full, and/or mount /var/log on a separate partition so that when it gets full, it doesn't clobber the other services writing out their config files and/or backing store for stateful information. I think that the separate-partition-for-/var/log hack will at least partition the problem (pun intended) so that the rest of the machine can keep running and providing services. Either that or I increase the disk capacity to 32G or 64G. But that seems like overkill for a networking appliance. Maybe the logs should be pushed off to syslog running on something else.
Thanks for confirming the problem Aergan.
I know this is a silly question but how are you guys bringing up a shell window? Are you using ssh or connecting to the serial console?
-
This happened to me when I was a bit overzealous in what protection modules I wanted loaded via suricata. Though I never did figure out which one was the main offender after unselecting and selecting only the ones I was certain I needed the problem went away as the log is rotated out every 7 days iirc?
For console I usually login via putty.
-
Thank you. I presume you are using putty for its ssh capability.
But if OPNsense is intended to be a networking appliance, apparently it needs a bit more work on self-preservation. I can understand trap-door-ing yourself out of the system during configuration, mostly of packet filters, but once a system is running, it should stay running under all conditions until either power or the hardware fails. Reliability is a key watchword for networking appliances. We need to put things in places where we don't have our fingers on the reset button. And I know that getting up a 2AM to drive 100mi to press the reset button on one of my firewalls is not going to leave me in a good mood. ;)
-
I am going to rebuild my system on the 16G miniSD card. This time I am going to create a partition for /var/log to prevent log overflow from killing OPNsense. I am not familiar with how and where filesystems are mounted during the boot process. Could someone give me a pointer?
Yes, I know, I could figure this out for myself after a couple hours of research but I'm feeling time-crunched and I bet someone could tell me in about 30 seconds.
Thanks!
-
I had around 50GB available and /var/log/suricata maked most of that space out.. I sure hope this can be sorted out by the 6.X release.