Everything works for 5 days, then "Default deny / state violation rule"

[c3101]

Hi everyone

I've got OPNsense 23.7.4 setup on proxmox, WAN is a physical interface, and 5 VLANs internally.

All is good in the world, when without reason (as far as I can find), everything stops working.
No more DHCP.
No more routing.
No nothing.

Promox still seems to be accessible, and all the VM's and containers seems to be behaving. Things in the same VLAN can talk to each other.

OPSsense though is inaccessible via the network.
Rebooting the OPNsense VM brings everything back, and things starts to work again,
Looking in the firewall log, there are plenty of "Default deny / state violation rule" messages.
But how can everything, at all once, be out of state?

I have no idea where to start looking to try and fix this.
It's now the 2nd time this week. Kinda anoying.