system.log filling up with empty configctl messages

[33b]

I'm on OPNsense 21.1.9_1-amd64. I noticed a sudden increase in disk space usage on the dashboard. I found an anomoly in the system.log from two days ago, which seemed to resolve itself without a reboot before I noticed (uptime is 20 days currently).

Code Select Expand


root@opn:/var/log/system # ls -lF
total 52649044
-rw-------  1 root  wheel         1368 Jul 23 21:36 system_20210723.log
-rw-------  1 root  wheel         1368 Jul 24 21:37 system_20210724.log
-rw-------  1 root  wheel         1364 Jul 25 21:38 system_20210725.log
-rw-------  1 root  wheel         1362 Jul 26 21:39 system_20210726.log
-rw-------  1 root  wheel  46888471248 Jul 27 23:28 system_20210727.log
-rw-------  1 root  wheel   7010838657 Jul 28 21:41 system_20210728.log
-rw-------  1 root  wheel         1536 Jul 29 13:56 system_20210729.log


Whoa, yeah... that's quite an increase! Let's take a closer look:

Code Select Expand


# tail system_20210727.log
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:
Jul 27 23:28:27 opn.localdomain configctl[40506]: event @ 1627449996.77 msg:


... but how many of those lines are there?

Code Select Expand


# grep configctl system_20210727.log | wc -l
601134222


Wow, over 600 million. And I have no idea what they're about, particularly with the "msg" field being empty.

Any thoughts? How would I go about debugging this further?

[33b]

P.S. I realized after writing this that the timing on July 27th suggests this occurred around the upgrade to 21.1.9 from 21.1.8.

[franco]

Something in the combination of Python and Syslog-ng update of 21.1.9... Not entirely sure what went on there and how to address this from our side. We will be looking at this combination more carefully in future stable updates.


Cheers,
Franco