I'm on OPNsense 21.1.9_1-amd64. I noticed a sudden increase in disk space usage on the dashboard. I found an anomoly in the system.log from two days ago, which seemed to resolve itself without a reboot before I noticed (uptime is 20 days currently).
root@opn:/var/log/system # ls -lF
total 52649044
-rw------- 1 root wheel 1368 Jul 23 21:36 system_20210723.log
-rw------- 1 root wheel 1368 Jul 24 21:37 system_20210724.log
-rw------- 1 root wheel 1364 Jul 25 21:38 system_20210725.log
-rw------- 1 root wheel 1362 Jul 26 21:39 system_20210726.log
-rw------- 1 root wheel 46888471248 Jul 27 23:28 system_20210727.log
-rw------- 1 root wheel 7010838657 Jul 28 21:41 system_20210728.log
-rw------- 1 root wheel 1536 Jul 29 13:56 system_20210729.log
Whoa, yeah... that's quite an increase! Let's take a closer look:
# tail system_20210727.log
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
... but how many of those lines are there?
# grep configctl system_20210727.log | wc -l
601134222
Wow, over 600 million. And I have no idea what they're about, particularly with the "msg" field being empty.
Any thoughts? How would I go about debugging this further?
P.S. I realized after writing this that the timing on July 27th suggests this occurred around the upgrade to 21.1.9 from 21.1.8.
Something in the combination of Python and Syslog-ng update of 21.1.9... Not entirely sure what went on there and how to address this from our side. We will be looking at this combination more carefully in future stable updates.
Cheers,
Franco