OPNsense Forum

Archive => 21.1 Legacy Series => Topic started by: 33b on July 29, 2021, 11:12:00 pm

Title: system.log filling up with empty configctl messages
Post by: 33b on July 29, 2021, 11:12:00 pm

I'm on OPNsense 21.1.9_1-amd64. I noticed a sudden increase in disk space usage on the dashboard. I found an anomoly in the system.log from two days ago, which seemed to resolve itself without a reboot before I noticed (uptime is 20 days currently).

Code: [Select]

root@opn:/var/log/system # ls -lF
total 52649044
-rw-------  1 root  wheel         1368 Jul 23 21:36 system_20210723.log
-rw-------  1 root  wheel         1368 Jul 24 21:37 system_20210724.log
-rw-------  1 root  wheel         1364 Jul 25 21:38 system_20210725.log
-rw-------  1 root  wheel         1362 Jul 26 21:39 system_20210726.log
-rw-------  1 root  wheel  46888471248 Jul 27 23:28 system_20210727.log
-rw-------  1 root  wheel   7010838657 Jul 28 21:41 system_20210728.log
-rw-------  1 root  wheel         1536 Jul 29 13:56 system_20210729.log

Whoa, yeah... that's quite an increase! Let's take a closer look:

Code: [Select]

# tail system_20210727.log
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:

... but how many of those lines are there?

Code: [Select]

# grep configctl system_20210727.log | wc -l
601134222

Wow, over 600 million. And I have no idea what they're about, particularly with the "msg" field being empty.

Any thoughts? How would I go about debugging this further?

Title: Re: system.log filling up with empty configctl messages
Post by: 33b on July 30, 2021, 05:47:20 am

P.S. I realized after writing this that the timing on July 27th suggests this occurred around the upgrade to 21.1.9 from 21.1.8.

Title: Re: system.log filling up with empty configctl messages
Post by: franco on July 30, 2021, 09:54:37 am

Something in the combination of Python and Syslog-ng update of 21.1.9... Not entirely sure what went on there and how to address this from our side. We will be looking at this combination more carefully in future stable updates.


Cheers,
Franco