[SOLVED] site-to-site OpenVPN Help

[Arthur Kay]

I have two VMs with OPNSense installed on them. One is at home, running in Hyper-V. The Hyper-V OPNSense instance is fed the apartment complex's ethernet jack as a private interface (its WAN port). It serves out on the main virtual switch, which is in turn connected to a real switch (all one subnet). This works perfectly. All the VMs and physical machines can talk to one another, and they all have internet access.

My second OPNSense instance runs on a VPS via Vultr. Its LAN link is their "private network" feature that spawns a virtual switch between VPS at the same datacenter. My other VPS have their WAN interfaces disabled, and this also works as expected. NAT rules for forwarding work, as do the VPS' internet connections.

The issue arises with connecting the two. I want to route ALL traffic via the cloud OPNSense instance, using my home WAN-link *only* to connect to the VPN. That's a simple enough routing rule to set up in theory, and I'm fine there (set a static route to the cloud OPNSense box over the DHCP gateway, turn off the default route). I have to use OpenVPN because I cannot forward ports here, and I want a few open to the world. The idea is to NAT them from the VPS' public IP to the home VM / machine's private IP. I followed various pfSense tutorials for setting up such a VPN. I followed https://doc.pfsense.org/index.php/Routing_internet_traffic_through_a_site-to-site_OpenVPN-connection_in_PfSense_2.1 exactly and what I got is that the local box (10.11.11.10/24) can ping the remote box (10.11.12.13/24) but not vice versa, and all the devices on the switch are happily routing over the public internet, even though I've explicitly told the VPN client otherwise on the local OPNSense box. Does that option not work with OPNSense yet? If it worked, surely the local machines would have no connectivity, vs working public internet?

I am currently attempting a layer 3 tunneled connection, but would vastly prefer to do layer 2 (and then build vlanned subnets later). I had layer 2 set up and half-working, where some sites would load, others wouldn't, and downloads (Steam, linode / vpsdime / digitalocean test files) would run at 1-20KB/s (but streaming 4K off youtube worked perfectly...). Both connections involved were/are 100Mbit. I've about given up on setting this up myself. How would I properly configure a layer 2 site-to-site OpenVPN (IPSec site-to-site requires port forwarding, which I can't do) with one of the sites routing *all* internet traffic via the other in OPNSense?

EDIT: Went back to layer 2, bridged with the tap adapter on both sides via web UI, re-subnetted to 10.20.30.40/24 (DHCP on the cloud side). I can reach the cloud OPNSense WebUI from the local net now, using its LAN address.

EDIT2: And I'm right back where I was when I rage-wiped both VMs. Set my machine's gateway to the cloud OPNSense instance and downloads are going 20kb/s while youtube works happily in 4K.

ED3: Vultr recommends an MTU of 1450 for their private network, and I have this set on the LAN interface of the remote OPNSense VM and on the lan interfaces of the other VPS. Is this the cause? Not sure where to go with that or how MTU works, especially over a VPN.

[Arthur Kay]

Getting close to figuring this out ... the issue is server-side, with the configuration of the VPN/bridge.

Just the act of turning on the VPN service (with tap bridged to LAN) puts the connectivity for the cloud VMs in the toilet. No clients connected. Home gateway not even powered on.

[pierrefrancois]

Hi Arthur,

If I understand correctly, your need would be to basically use the Vultr OPNsense as your internet gateway for both the other VM in Vultr and your home network, your OPNsense at home simply beeing a transparent bridge.
If you got issue turning on the VPN service I would suspect some IP/routing issue.
What is the IP/subnet you use on the LAN and the OpenVPN interface? are you sure there's no overlap?

[Arthur Kay]

With a bridge, wouldn't it be the same subnet since DHCP / broadcasts would come over the layer 2 from the Vultr OPNSense? Both are 10.20.30.0/24 right now but I could VLAN + subnet if that's a better architecture, and set up a DHCP relay on the local OPNSense. Would UPNP still work for automatic inbound NAT?

[pierrefrancois]

The LAN IP of both OPNsense should be in this subnet but not the OpenVPN interface. For example you can use 10.250.0.0/30 with 10.250.0.1 on your Vultr OPNsense and 10.250.0.2 on your home.
As you use TAP interface you should configure the IP on both side like this
in your Vultr OpenVPN : ifconfig 10.250.0.1 255.255.255.252
in your home OpenVPN : ifconfig 10.250.0.2 255.255.255.252

if you could post a bit more information on the IP used on your setup it will be easier to help.

I don't understand what you mean by this "Would UPNP still work for automatic inbound NAT?"

[Arthur Kay]

UPNP is what enables devices / software to request a port forward in certain port ranges, no? A security liability, but very convenient if consumer devices are on the LAN.

Great point about assigning the TAP adapters some other IP range. I'll give that a shot. Currently both networks are 10.20.30.0/24 with no DHCP on the local OPNsense

[pierrefrancois]

I know what is UPNP, I just don't understand what is the link with the issue of establishing the openvpn connection.
Don't forget, rou LAN should remain in 10.20.30.0/24
Your OpenVPN link should use the 10.250.0.0/30 or any other /30 subnet that is not in your LAN subnet
Keep in mind that this is only working if you use TAP interface.

[Arthur Kay]

Issue is resolved. I set it up exactly as it was before where I had a layer 2 bridge on one subnet, only I disabled all hardware acceleration on the Vultr VPS instance of OPNsense. Speedtest works just dandy, though it got better speed in SoftEther (100mbit vs ~50)

So +1 for the request to build SoftEther into OPNsense as a replacement for OpenVPN?

[franco]

A little late to the party I guess. Nice to see the issue resolved. SoftEther package will make it's way into 15.7.12, but will only available as a package until we find the time to provide a plugin with GUI and backend parts as well.

At least: no more manual building SoftEther on OPNsense from now on.