After upgrade to 23.7 and OPNsense 23.7.1_3-amd64 High disk write activity

[Albert38]

I'm running OPNsense already for a while in ProxMox. I noticed after the upgrade to 23.7 on august 6th an increase in disk write activity and a few day's after this upgrade more updates from OPNsense came august 9th and the disk write activity increased even more till roughly 350K Is this normal?

I also see that the use diskspace has almost doubled after the upgrade.

It appears OPNsense is running normal at the moment.

[aes]

I've noticed similar situation except over longer time period. My /var/log and /tmp are both configured for RAM disk and I've minimized logging in general, yet the disk is filling up over time and I can't figure out why.
I had to reinstall opnsense entirely at one point because disk was completely full thanks to some python module directory, and I was unable to extend the volumes as I'm not super familiar with bsd underneath the hood. You can see the reinstall time at approx 08-20 on "disk io year average" section in my attachment and how it has started steadily climbing since then again.

I had none of these problems when using pfsense :/

[franco]

That sawtooth pattern looks like local NetFlow capturing. Are you using insights reporting?


Cheers,
Franco

[Albert38]

Hi Franco,

On my side it is still increasing see attached graphics. I do not use Netflow capturing.

b.t.w. in /var/log I can't find any big log files except for /var/log/filter in there for every day a file in between 50 and 100mb is written, don't know if this is normal. I found a way to decrease the amount of information written bij the firewall, let's see if that changes the behavior.

[franco]

I was replying to aes specifically. Not sure about your pattern. What processes are running that use a lot of memory/cpu?

#  top -a | head -n20


Cheers,
Franco

[Albert38]

Hi Franco on my side no change to the behavior after disabling firewall logs.

the result of : top -a | head -n20
71375 root          8  20    0   272M   205M kqread   1  91:57   0.00% /usr/local/bin/python3 /usr/local/opnsense/scripts/unbound/logger.py (python3.9)
67686 unbound       4  20    0   111M    55M kqread   2   5:43   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf
57328 root          1  20    0    18M  7524K select   2   5:14   0.00% /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
70371 root          1  20    0    25M    15M select   3   2:37   0.00% /usr/local/bin/python3 /usr/local/opnsense/scripts/dhcp/unbound_watcher.py --domain 1.ombouw.com (pyth
  392 root          1  20    0    11M  1504K select   0   2:00   0.00% /sbin/devd
26715 root          1  20    0    21M  7092K select   1   1:39   0.00% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
49718 root          1  20    0    23M  6704K select   0   0:30   0.00% /usr/local/sbin/mpd5 -b -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient
54240 root          1  52    0    13M  2500K nanslp   2   0:14   0.00% /usr/sbin/cron -s
13200 root          3  20    0    49M    14M kqread   1   0:03   0.00% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
96561 root          1  20    0    23M    12M select   0   0:03   0.00% /usr/local/bin/python3 /usr/local/sbin/configctl -e -t 0.5 system event config_changed (python3.9)
96564 root          1  20    0    23M    12M select   3   0:03   0.00% /usr/local/bin/python3 /usr/local/opnsense/scripts/syslog/lockout_handler (python3.9)
32929 root          1  21    0    13M  2580K wait     0   0:03   0.00% /bin/sh /var/db/rrd/updaterrd.sh
90561 root          1  20    0    84M    47M accept   0   0:02   0.00% /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.9)

[franco]

At first glance you enabled Unbound DNS reporting?


Cheers,
Franco

[Albert38]

It looks like Unbound DNS reporting is on I think. It is showing information in Reporting: Unbound DNS
I have searched a bit but how to disable this? And is this new? Was it not in the version before 23.7? I do not recall I have enabled features when I did the update?

Found and disabled Unbound DNS reporting, overlooked the setting.

I can happily confirm disabling Unbound DNS reporting made my disk IO drop back to a level I was used to see. Thanks franco for pointing me in the right direction.

[franco]

It's an opt-in feature. Maybe you tried it and forgot to turn it on. No big deal. Happy this looks reasonable now.


Cheers,
Franco

[aes]

That sawtooth pattern looks like local NetFlow capturing. Are you using insights reporting?


Cheers,
Franco

Seems like I did have NetFlow capturing turned on, it was also completely misconfigured.. 
But more importantly, I also had Unbound DNS reporting turned on and my IO usage went down on the spot after I turned that off.

Edit: Also after I hit 'Reset DNS data', that got rid of over 2gb of data. Finally I know what was piling up
That said, having that data available was quite useful so would be nice to have some better data management tied to said reporting in the future.