Massive spike in states every day around 00:15am killing WAN connection

[Phiolin]

I've recently switched from pfSense to OPNsense, running 21.1.7_1-amd64.
For the last days, every night at 00:15am I see a huge spike in state entries (like hundreds of thousands) which kills the connection to my DSL modem and hence causes WAN connectivity to go dark until I reboot the firewall in the morning.
Currently struggling to find out what causes this, as there is no corresponding traffic spike on any interface, the only other indication is a spike of outpass6 packets on WAN. From what I can see it looks like these connections are coming from the firewall itself, which makes sense given nothing else has changed on my network in terms of services, so there's no reason there should be more traffic from any of the running clients now that I switched to OPNsense, as with pfSense this never happened.

For what it's worth, I run sensei on LAN and suricata on LAN (but not in IPS mode) and as I see a huge number of DNS related connections outgoing during the concerning time, it might be worth mentioning that I also have some DNSBL lists configured in Unbound. No custom ones though, just selected a couple of the blocklist.site pre-configured ones. The DNS queries are inconclusive though, it seems to be mostly unbound contacting various root-servers, so maybe it is just trying to resolve a great amount of queries, for which however I have not seen any client-queries coming through. I have query log enabled and pipe the whole stuff into pfElk, so at least I have some visibility, but there's no real indicator of what would be going on.

In pfElk I can see a spike of firewall events between 00:15 and 00:16 which are all "pass" events, so nothing is getting blocked. Looking up the IP addresses, it is a large amount of IPv6 destination IPs (which explains the outpass6 packets spike) and if I search for these, it is all *.root-servers.net, so I guess it's really unbound related, the question is however what triggers it.
Within the time window where Unbound seems to generate 18.000 firewall entries for various root servers, I only see 180 client-side DNS requests in the query log, so those numbers don't match up, unless Unbound is querying root-servers for a completely different reason.
The firewall is running smooth throughout the whole day otherwise, just shortly after midnight it seems to go haywire.

Anyone have any ideas?


 

[Fright]

Hi
maybe there are interesting entries in the general log before this?
any cron jobs for this time?

[Phiolin]

I found it, took a while though.
I'm running *sense virtualized on Proxmox, and while switching from pfSense to OPNsense, I missed to disable the VM backup for pfSense. Every night at 00:15 Proxmox would try to spin up the pfSense VM for backup and as I PCI passthrough the network adapter for WAN, that would mess with the passthrough and the OPNsense VM would lose access to the device. Everything else was just symptoms... works fine now.