Suricata crashed router by filling the drive with logs

[LOENS2]

Hi there.

Today my OPNsense box crashed due to no space left on the drive (256 GB SSD).

Upon closer inspection I noticed, that Suricata filled up the /var/logs/suricata with a total of about 4.4 TB of log files, which obviously filled up my drive. After deleting all of the Suricata log files the box worked again. How can this happen and how can I prevent that from happening again? I'll attach some screenshots.

The result of "find /var/logs -size +10M -ls":

Code: [Select]

     2596   2321 -rw-------   1 root     root     10496616 Jun 14 12:58 ./flowd.log.000007
      255   6489 -rw-------   1 root     root     36355528 Jun 11 01:59 ./filter/filter_20230610.log
     2515  25713 -rw-------   1 root     root     167356099 Jun  7 01:59 ./filter/filter_20230606.log
      685  37965 -rw-------   1 root     root     229062871 Jun  2 01:59 ./filter/filter_20230601.log
    18455  32753 -rw-------   1 root     root     212272908 Mai 19 01:59 ./filter/filter_20230518.log
     3639  27049 -rw-------   1 root     root     190829774 Jun  9 01:59 ./filter/filter_20230608.log
    20673  23645 -rw-------   1 root     root     154170208 Mai 24 01:59 ./filter/filter_20230523.log
    20605  21253 -rw-------   1 root     root     137660555 Mai 25 01:59 ./filter/filter_20230524.log
      917   6593 -rw-------   1 root     root      37149315 Jun 12 01:59 ./filter/filter_20230611.log
    21434  21533 -rw-------   1 root     root     139952405 Mai 26 01:59 ./filter/filter_20230525.log
    20279  23049 -rw-------   1 root     root     150245976 Mai 23 01:59 ./filter/filter_20230522.log
       96   6077 -rw-------   1 root     root      33453305 Jun 10 01:59 ./filter/filter_20230609.log
    18517  23837 -rw-------   1 root     root     155406447 Mai 20 01:59 ./filter/filter_20230519.log
    17927  30209 -rw-------   1 root     root     193107374 Mai 18 01:59 ./filter/filter_20230517.log
     3403  27993 -rw-------   1 root     root     186206910 Jun  8 01:59 ./filter/filter_20230607.log
    19657  26261 -rw-------   1 root     root     171177735 Mai 22 01:59 ./filter/filter_20230521.log
      476  24393 -rw-------   1 root     root     159002768 Mai 27 01:59 ./filter/filter_20230526.log
      999  23605 -rw-------   1 root     root     153357876 Mai 29 01:59 ./filter/filter_20230528.log
     1631  25921 -rw-------   1 root     root     168651020 Jun  5 01:59 ./filter/filter_20230604.log
     1826  23401 -rw-------   1 root     root     152466766 Jun  4 01:59 ./filter/filter_20230603.log
     1470   6521 -rw-------   1 root     root      35676822 Jun 13 01:59 ./filter/filter_20230612.log
     2826   5393 -rw-------   1 root     root      29658805 Jun 16 01:55 ./filter/filter_20230615.log
     2155  23725 -rw-------   1 root     root     154645439 Mai 31 01:59 ./filter/filter_20230530.log
     1317  25933 -rw-------   1 root     root     168540286 Jun  3 01:59 ./filter/filter_20230602.log
     2493  24465 -rw-------   1 root     root     159374873 Jun  6 01:59 ./filter/filter_20230605.log
     1648  24005 -rw-------   1 root     root     156458972 Mai 30 01:59 ./filter/filter_20230529.log
      613  24769 -rw-------   1 root     root     161317388 Mai 28 01:59 ./filter/filter_20230527.log
    19366  25061 -rw-------   1 root     root     163379154 Mai 21 01:59 ./filter/filter_20230520.log
      180  32045 -rw-------   1 root     root     207209373 Jun  1 01:59 ./filter/filter_20230531.log
     2186   6145 -rw-------   1 root     root      33609659 Jun 15 01:59 ./filter/filter_20230614.log
     1769   6145 -rw-------   1 root     root      33731614 Jun 14 01:59 ./filter/filter_20230613.log
     2184   2429 -rw-------   1 root     root      10492648 Jun 13 19:38 ./flowd.log.000010
     3016   2381 -rw-------   1 root     root      10509104 Jun 15 11:36 ./flowd.log.000003
     2389   2517 -rw-------   1 root     root      10495520 Jun 14 00:36 ./flowd.log.000009
     3346   6429 -rw-------   1 root     root      20233128 Jun 16 10:39 ./flowd.log
     2961   2425 -rw-------   1 root     root      10507228 Jun 15 05:37 ./flowd.log.000004
     2209   2397 -rw-------   1 root     root      10500556 Jun 14 18:48 ./flowd.log.000006
     3106   2593 -rw-------   1 root     root      10504972 Jun 15 23:14 ./flowd.log.000001
     2040   2497 -rw-------   1 root     root      10498920 Jun 15 00:01 ./flowd.log.000005
     3663 637297 -rwx------   1 root     root     20480905817 Jun  8 23:57 ./suricata/suricata_20230608.log
      425 34661697 -rw-------   1 root     root     1113869720977 Jun 11 02:00 ./suricata/suricata_20230610.log
      119 25591289 -rw-------   1 root     root      822438998271 Jun 10 02:00 ./suricata/suricata_20230609.log
     2933    49157 -rw-------   1 root     root        1568197361 Jun 16 10:39 ./suricata/suricata_20230616.log
      911 34620437 -rw-------   1 root     root     1112538559783 Jun 12 02:00 ./suricata/suricata_20230611.log
     1577 34711293 -rw-------   1 root     root     1115442512172 Jun 13 02:00 ./suricata/suricata_20230612.log
     2979 32021273 -rw-------   1 root     root     1028300789488 Jun 16 02:00 ./suricata/suricata_20230615.log
     2425 34646789 -rw-------   1 root     root     1113319976526 Jun 15 02:00 ./suricata/suricata_20230614.log
     1848 34582617 -rw-------   1 root     root     1111373179464 Jun 14 02:00 ./suricata/suricata_20230613.log
     2424     2413 -rw-------   1 root     root          10488392 Jun 14 06:20 ./flowd.log.000008
     3035     2285 -rw-------   1 root     root          10495064 Jun 15 18:20 ./flowd.log.000002


[9axqe]

Similar issue for me, I had flowd.log at 48GB after less than a month. It clearly started after enabling IDS/IPS.

Did you make any progress on this? Currently I am limiting my logs to a single day and still manage to get above 50GB of logs for single household...

[LOENS2]

No, unfortunately I didn't find a solution yet. I also limited the logs. As I wrote in my previous post, I had 4.4 TB of data, but I only have a 256GB drive. I run my OPNsense setup in a single household, too, but I own a few servers, maybe that's where the problem is coming from for me. Finding out the problem wasn't easy for me, as my OPNsense wouldn't even start anymore and I had to delete the logs on another PC.

[9axqe]

thanks. Same for me, I had to connect to console using mini-USB to delete logs to fix it, opnsense couldn't even boot...