Wireguard VPN Does Not Work When Protectli Vault Is Plugged Directly Into Modem

[opnsense1]

I have my Wireguard Proton VPN working well with my default config of Coaxial > ISP Modem > ISP Router (Not in bridged mode as other residents use the ISP provided network) > Protectli Vault > My Router.

It's not ideal for me but double NAT isn't as much of an issue as I don't want the devices on the ISP router to interact with my devices and vice versa at all.

I do find it odd though that when testing removing the ISP Router and just plugging the Protectli Vault into the modem, after some rebooting, I get green status dots on all 3 of my gateways including the VPN one, but when checking my IP address on https://ip.me, it shows my real public IP and location rather than the VPN.

Any ideas why this would happen? Seems like odd behaviour to me, especially because I have the killswitch enabled so connections that aren't through my VPN shouldn't be allowed at all and this works correctly when I am plugged into the ISP router. (Confirmed by not having internet for a few days because I had no idea Proton changed the endpoint address on me 🤦)