NTP server - no active peers - problem

[qinohe]

Have an issue using timeservers.
Setting a pool of timeservers, it doesn't matter what I choose.
As an example I used the NL & OPNsense pool - 0-3.nl.pool.ntp.org & 0-3.opnsense.pool.ntp.org
But using single timeservers hosted by some uni's(NL) leads to the same outcome.

OPNsense is hosted by a (local) Proxmox server which runs as it should.
Proxmox is bridged behind OPNsense in the picture of OPNsense as a router.
This also leads to no problems and runs very stable.

NTP is the only service giving me a headache for it looses it's connection after an undefined amount of time.
A message is displayed on the dashboard: No active peers available.

Mostly the NTP server will find a new peer and the time will be synchronized again on the network.
But, if the 'non active peer' situation takes too long I will be notified by the 'Check_MK' - read Nagios - server there are problems with time syncing.
Check_mk often gives a warning the service is 'flapping' - meaning it's condition is not stable and changes quickly (not measurable but quite indicative)

Because OPNsense runs on Proxmox there is no 'real' hardware clock and the clock is only run by the virtual processor.
You can imagine the offset goes haywire within a few hours, the clock can be off by (many) minutes.

Resetting the NTP server most of the times solves the problem, but not always!
One of the problems I have and is really annoying is OTP authentication.
My codes are on my phone which is on a 4G network. I think you see the problem...
Had to stop using OTP for most apps cause it became a to unstable situation.

Anyway, this has ran stable for almost a year, but I'm having problems with it for a while now - I can't remember exactly when it started, sorry!
Manually resetting the NTP server may be needed a few times per day.

Is there a way to figure out what stops the NTP server from using perfectly fine pools.

Many thanks in advance if you can help me figure this out