After upgrade to 23.7 and OPNsense 23.7.1_3-amd64 High disk write activity

[Albert38]

I'm running OPNsense already for a while in ProxMox. I noticed after the upgrade to 23.7 on august 6th an increase in disk write activity and a few day's after this upgrade more updates from OPNsense came august 9th and the disk write activity increased even more till roughly 350K Is this normal?

I also see that the use diskspace has almost doubled after the upgrade.

It appears OPNsense is running normal at the moment.

[aes]

I've noticed similar situation except over longer time period. My /var/log and /tmp are both configured for RAM disk and I've minimized logging in general, yet the disk is filling up over time and I can't figure out why.
I had to reinstall opnsense entirely at one point because disk was completely full thanks to some python module directory, and I was unable to extend the volumes as I'm not super familiar with bsd underneath the hood. You can see the reinstall time at approx 08-20 on "disk io year average" section in my attachment and how it has started steadily climbing since then again.

I had none of these problems when using pfsense :/

[franco]

That sawtooth pattern looks like local NetFlow capturing. Are you using insights reporting?


Cheers,
Franco

[Albert38]

Hi Franco,

On my side it is still increasing see attached graphics. I do not use Netflow capturing.

b.t.w. in /var/log I can't find any big log files except for /var/log/filter in there for every day a file in between 50 and 100mb is written, don't know if this is normal. I found a way to decrease the amount of information written bij the firewall, let's see if that changes the behavior.

[franco]

I was replying to aes specifically. Not sure about your pattern. What processes are running that use a lot of memory/cpu?

#  top -a | head -n20


Cheers,
Franco

[Albert38]

Hi Franco on my side no change to the behavior after disabling firewall logs.

the result of : top -a | head -n20
71375 root          8  20    0   272M   205M kqread   1  91:57   0.00% /usr/local/bin/python3 /usr/local/opnsense/scripts/unbound/logger.py (python3.9)
67686 unbound       4  20    0   111M    55M kqread   2   5:43   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf
57328 root          1  20    0    18M  7524K select   2   5:14   0.00% /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
70371 root          1  20    0    25M    15M select   3   2:37   0.00% /usr/local/bin/python3 /usr/local/opnsense/scripts/dhcp/unbound_watcher.py --domain 1.ombouw.com (pyth
  392 root          1  20    0    11M  1504K select   0   2:00   0.00% /sbin/devd
26715 root          1  20    0    21M  7092K select   1   1:39   0.00% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf
49718 root          1  20    0    23M  6704K select   0   0:30   0.00% /usr/local/sbin/mpd5 -b -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient
54240 root          1  52    0    13M  2500K nanslp   2   0:14   0.00% /usr/sbin/cron -s
13200 root          3  20    0    49M    14M kqread   1   0:03   0.00% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
96561 root          1  20    0    23M    12M select   0   0:03   0.00% /usr/local/bin/python3 /usr/local/sbin/configctl -e -t 0.5 system event config_changed (python3.9)
96564 root          1  20    0    23M    12M select   3   0:03   0.00% /usr/local/bin/python3 /usr/local/opnsense/scripts/syslog/lockout_handler (python3.9)
32929 root          1  21    0    13M  2580K wait     0   0:03   0.00% /bin/sh /var/db/rrd/updaterrd.sh
90561 root          1  20    0    84M    47M accept   0   0:02   0.00% /usr/local/bin/python3 /usr/local/opnsense/service/configd.py console (python3.9)

[franco]

At first glance you enabled Unbound DNS reporting?


Cheers,
Franco

[Albert38]

It looks like Unbound DNS reporting is on I think. It is showing information in Reporting: Unbound DNS
I have searched a bit but how to disable this? And is this new? Was it not in the version before 23.7? I do not recall I have enabled features when I did the update?

Found and disabled Unbound DNS reporting, overlooked the setting.

I can happily confirm disabling Unbound DNS reporting made my disk IO drop back to a level I was used to see. Thanks franco for pointing me in the right direction.

[franco]

It's an opt-in feature. Maybe you tried it and forgot to turn it on. No big deal. Happy this looks reasonable now.


Cheers,
Franco

[aes]

That sawtooth pattern looks like local NetFlow capturing. Are you using insights reporting?


Cheers,
Franco

Seems like I did have NetFlow capturing turned on, it was also completely misconfigured..  :-X
But more importantly, I also had Unbound DNS reporting turned on and my IO usage went down on the spot after I turned that off.

Edit: Also after I hit 'Reset DNS data', that got rid of over 2gb of data. Finally I know what was piling up :)
That said, having that data available was quite useful so would be nice to have some better data management tied to said reporting in the future.