Found the culprit: There was a blackhole /56 route that matched the /64 of the VPN IPs. Apparently when a blackhole route exists, the traffic selector does not have the chance to match the traffic.
Deleting the blackhole route fixed the issue. Is this a known thing?
Deleting the blackhole route fixed the issue. Is this a known thing?