Any small device which can run OpenVPN.A raspberry Pi for example but depends on the throughput you need
It's no different from any other site-to-site OpenVPN setup, so possibly exlude the "raspberry pi" keywords from your searches.You will ned to to an inbound port forwarding for 1194/udp or whatever you prefer on the NAT gateway/router in that network but apart from that the setup should work with any generic "howto" document.
If the RPi is the client and initiates the tunnel, why do I need a port forward on the NAT device it sits behind?
Either the client-side NAT is missing
The RPi gets NAT'ed and there doesn't appear to be any documentation that I can find that spells out whether the router has to explicitly support NAT for VPN devices or not. If so, then this changes everything as I will ultimately end up swapping out the router with one that has OpenVPN support directly in it and this becomes a moot point.
rks with standard NAT without any special configuration as long as that is only on one side - which needs to initiate the connection. That's why there's no documentation. Nothing to see here, just works.