On the Server side when you add endpoint, only enter the remote Tunnel IP with /32 there. On endpoint itself you can add 0.0.0.0/0 as allowed ip
To make things worse, there's a ton of misinformation out there.