High Latency - every 8 hours

[gafrol]

I am observing high-latency pings to the local wired Interface of the Opnsense firewall from the LAN. Starting at 4 am (CET) latency climbs to 2500 for about 2-3 minutes, repeating every 8 hours.

I can replicate the issue on every interface. For troubleshooting purposes, I have configured a new interface put a Windows machine in the newly configured Network, and connected it with an ethernet cable directly to the Opnsense firewall, with no switch in between. Just the Win PC -- ethernet cable - Opnsense. I don't see any interface errors or collisions in the statistics.

Again the issue repeats every 8 hours, starting at 4am. Trying to understand what is causing this behavior.

[gafrol]

in /var/log/system/latest.log I see these log entries every time this happens.

<13>1 2023-11-07T04:01:09+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="1"] vacuum dst_port_000300.sqlite
<13>1 2023-11-07T04:01:09+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="2"] vacuum dst_port_003600.sqlite
<13>1 2023-11-07T04:01:09+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="3"] vacuum dst_port_086400.sqlite
<13>1 2023-11-07T04:05:20+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="1"] vacuum src_addr_details_086400.sqlite
<13>1 2023-11-07T04:08:13+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="1"] vacuum src_addr_000300.sqlite
<13>1 2023-11-07T04:08:13+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="2"] vacuum src_addr_003600.sqlite
<13>1 2023-11-07T04:08:13+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="3"] vacuum src_addr_086400.sqlite
<13>1 2023-11-07T04:09:29+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="1"] vacuum interface_000030.sqlite
<13>1 2023-11-07T04:09:30+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="2"] vacuum interface_000300.sqlite
<13>1 2023-11-07T04:09:31+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="3"] vacuum interface_003600.sqlite
<13>1 2023-11-07T04:09:31+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="4"] vacuum interface_086400.sqlite
<13>1 2023-11-07T04:09:31+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="5"] vacuum done




<13>1 2023-11-07T12:01:48+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="1"] vacuum dst_port_000300.sqlite
<13>1 2023-11-07T12:01:48+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="2"] vacuum dst_port_003600.sqlite
<13>1 2023-11-07T12:01:48+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="3"] vacuum dst_port_086400.sqlite
<13>1 2023-11-07T12:05:56+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="1"] vacuum src_addr_details_086400.sqlite
<13>1 2023-11-07T12:08:17+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="1"] vacuum src_addr_000300.sqlite
<13>1 2023-11-07T12:08:17+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="2"] vacuum src_addr_003600.sqlite
<13>1 2023-11-07T12:08:17+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="3"] vacuum src_addr_086400.sqlite
<13>1 2023-11-07T12:09:24+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="1"] vacuum interface_000030.sqlite
<13>1 2023-11-07T12:09:25+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="2"] vacuum interface_000300.sqlite
<13>1 2023-11-07T12:09:26+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="3"] vacuum interface_003600.sqlite
<13>1 2023-11-07T12:09:26+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="4"] vacuum interface_086400.sqlite
<13>1 2023-11-07T12:09:26+01:00 OPNsense.localdomain flowd_aggregate.py 75395 - [meta sequenceId="5"] vacuum done


Has this something to do with reporting?

[meyergru]

Obviously. If you disable flowd, there would be no need to flush the sqlite database.

[gafrol]

When switching off REPORTING: NETFLOW the observed high-latency disappears.