Multiple IPv6 IPs on WAN issues

[r3pek]

Hi guys. I'm trying to understand with my VPS provider with isn't my IPv6 setup working properly.

I'm assigned 5 /64 IPv6, which one is the main one and the other 4 are like aliases. To configure this on OPNSense, I've configured one directly on the interface using Static IPv6 and defining a custom GW for it, the others are defined on the Virtual IPs setting.
The symptoms I'm observing is that only when I ping the IPv6 GW from any of the source IPv6 is that the traffic actually "works", as in, I can reach the internet via IPv6. After 10-20s of stopping the ping to the GW, all traffic stops (from either of the 5 IPv6s).

What am I missing on my configuration or what did I do wrong here?