Clients randomly not working -- out of disk

[BrianLloyd]

I have begun to have a problem with my WiFi clients not being able to get assigned their configuration with DHCP. It started a couple days ago with my iPhone randomly not being able to access the net because it does not have an assigned IP address. I rebooted last night and everything came back. This evening the problem is back. First thing I noticed was that the FS is full.

/ (ufs): 109% used 14G/14G

WTF? What would be eating that much disk? This is a pretty basic installation. So off to the logs. The system log is full of:

kernel: pid 73581 (dhcpd), uid 136 inumber 1364898 on /mnt: filesystem full

and

kernel: pid 59843 (suricata), uid 0 inumber 1364877 on /mnt: filesystem full

So it stands to reason that dhcpd has a problem not being able to write out the leases. That would answer why the clients aren't getting assigned addresses. But I'm still at a loss as to what would consume all the disk and cause the problem in the first place. how to clear it out again.

Oh, and I stopped intrusion detection just in case. It does seem to be trying to write a lot of stuff to disk.

Help?

[franco]

We don't write /mnt as far as I know. Knowing Suricata it might have spammed /var/log, please provide the following output for further assistance.

# ls -lah /var/log
# ls -lah /var/log/squid
# ls -lah /var/log/suricata

[Aergan]

Think we have a winner for mine:

Code: [Select]

ls -lah /var/log/suricata
total 16589428
drwx------  2 root  wheel   512B Nov 27 00:00 .
drwxr-xr-x  6 root  wheel   1.5K Nov 27 00:00 ..
-rwx------  1 root  wheel     0B Nov 22 23:00 eve.json
-rwx------  1 root  wheel     0B Nov 22 23:00 eve.json.0
-rwx------  1 root  wheel   594B Nov 15 23:00 eve.json.1
-rwx------  1 root  wheel    16G Nov  8 23:00 eve.json.2
-rwx------  1 root  wheel   231K Nov  1 23:00 eve.json.3
-rwx------  1 root  wheel   8.8M Nov 27 08:16 stats.log
-rwx------  1 root  wheel    37M Nov 27 00:00 stats.log.0
-rwx------  1 root  wheel    32M Nov 26 00:00 stats.log.1
-rwx------  1 root  wheel    26M Nov 25 00:00 stats.log.2
-rwx------  1 root  wheel    37M Nov 24 00:00 stats.log.3
-rwx------  1 root  wheel   1.8M Nov 23 00:00 stats.log.4
-rwx------  1 root  wheel    43M Nov 22 00:00 stats.log.5
-rwx------  1 root  wheel    37M Nov 21 00:00 stats.log.6

Removed the offending file: eve.json.2

Gone down to:
/ (ufs): 4% used 1.1G/28G

[BrianLloyd]

Thanks. Well, the issue is moot for me at this point. Shortly after writing my original posting the web interface displayed an error message about there not being a config file instead of showing the relevant page. I (stupidly) rebooted the machine. OPNsense did not come back up and the LAN interface never responded to pings. I think I am going to have to rebuild the system. Fortunately for me I was able to fall back to my m0n0wall system and keep my network running.

Clearly there is some kind of problem with something filling up the disk storage. I suspect the right answer is to either run a utility to roll the logs and delete old ones when /var/log gets too full, and/or mount /var/log on a separate partition so that when it gets full, it doesn't clobber the other services writing out their config files and/or backing store for stateful information. I think that the separate-partition-for-/var/log hack will at least partition the problem (pun intended) so that the rest of the machine can keep running and providing services. Either that or I increase the disk capacity to 32G or 64G. But that seems like overkill for a networking appliance. Maybe the logs should be pushed off to syslog running on something else.

Thanks for confirming the problem Aergan.


I know this is a silly question but how are you guys bringing up a shell window? Are you using ssh or connecting to the serial console?

[Solaris17]

This happened to me when I was a bit overzealous in what protection modules I wanted loaded via suricata. Though I never did figure out which one was the main offender after unselecting and selecting only the ones I was certain I needed the problem went away as the log is rotated out every 7 days iirc?

For console I usually login via putty.

[BrianLloyd]

Thank you. I presume you are using putty for its ssh capability.

But if OPNsense is intended to be a networking appliance, apparently it needs a bit more work on self-preservation. I can understand trap-door-ing yourself out of the system during configuration, mostly of packet filters, but once a system is running, it should stay running under all conditions until either power or the hardware fails. Reliability is a key watchword for networking appliances. We need to put things in places where we don't have our fingers on the reset button. And I know that getting up a 2AM to drive 100mi to press the reset button on one of my firewalls is not going to leave me in a good mood. 

[BrianLloyd]

I am going to rebuild my system on the 16G miniSD card. This time I am going to create a partition for /var/log to prevent log overflow from killing OPNsense. I am not familiar with how and where filesystems are mounted during the boot process. Could someone give me a pointer?

Yes, I know, I could figure this out for myself after a couple hours of research but I'm feeling time-crunched and I bet someone could tell me in about 30 seconds.

Thanks!

[guest10459]

I had around 50GB available and /var/log/suricata maked most of that space out.. I sure hope this can be sorted out by the 6.X release.