...Any ideas?
There's a dedicated forum for VPN questions
What OPNsense version(s) are you using ?
Are you using Policy or Route based IPsec ?
How does your topology look like ?
Does Router A allows both UDP 500 and ESP traffic from destination B on WAN interface ?