NTP is an IPv6 lover. So, if you don't have a working IPv6 connection and you didn't disable IPv6, then it fails to work.
I wonder why my clock was drifting, though.