Always On VPN (Wireguard) Mobile Phone Clients - Connection issues internally

[foss-johnny]

Hi all,

I'm having issues with an Always-On-VPN setup I'm trying to get working for mobile clients using Wireguard VPN.

My goal is to always leave the Wireguard VPN On for all mobile phones, and iand for them to roam between the internal wi-fi and their 5g connection, all traffic for the mobile client should route through the Opnsense firewall.

Here's the toplogy.

192.168.1.x = Internal Wireless Subnet
172.16.1.x = Wireguard Subnet
10.0.1.x = File server Subnet

Currently if the mobile phone is connected to the 5G connection, everything is working fine.

However, when the mobile phone is connected to internal wifi, and the Wireguard connection is sucessfully established, I try to connect apps (file server), and I receive a "TIME_WAIT:TIME_WAIT" message in the session logs.

After reviewing the firewall traffic logs I can see that the traffic is allowed and "pass" status.

However, when comparing a trace route from the mobile to the file server when on the wifi and wg connected, it does not hit the wireguard gateway first, instead I see * * * .. for each hop.

Does anyone have a configuration like this working properly or know how to resolve?

Thanks!

[keeka]

Not sure where your issue lies but the way I have done this (for both openvpn and wireguard) is one destination NAT rule on the WAN, and another on the relevant lan interface. Both forwarding wireguard port to 127.0.0.1. I found that to be the most reliable way to get mobile/wifi roaming whilst only using WAN IP in any vpn client config. The WAN version of the port forward and fw rule filters src by my mobile provider's ASN.

[foss-johnny]

Not sure where your issue lies but the way I have done this (for both openvpn and wireguard) is one destination NAT rule on the WAN, and another on the relevant lan interface. Both forwarding wireguard port to 127.0.0.1. I found that to be the most reliable way to get mobile/wifi roaming whilst only using WAN IP in any vpn client config. The WAN version of the port forward and fw rule filters src by my mobile provider's ASN.

Thanks for sharing and explaining your setup Keeka.

One question - How is your DNS configured within wireguard on the client and in Unbound DNS and external DNS server (e.g. cloudflare)?

I'm use a DNS A record in Cloudflare that points to my WAN IP. Internally, when on the Wireless LAN, the mobile client resolves the Wireguard DNS name (e.g. vpn.domain.com) to my WAN IP address, I have a rule to allow the traffic from the wireless network to the WAN interface.

I don't have any destination NAT rules configured for Wireguard, because Wireguard is listenting on the WAN IP on UDP 51821 already.

Can you please help me understand how you have your DNS configured?

Will give your configuration a try.

p.s. Really like how you've filtered on ASN for the WAN rule

[foss-johnny]

When I am on the internal wi-fi, I connect to wireguard, and can see the tunnel is connected in the WG status page. The IP address of the client is from the internal wifi subnet.

However, when I ping the wireguard gateway IP address, I get patchy responses, one reply, followed by timeouts.

Sometimes I ping, and I don't even get one reply back.

For minute I thought I had a asymmetric route, but after taking a packet capture on all interfaces, I can see that when the ping traffic fails, there is not tcmpdump entry.

It's as if the traffic from the mobile client is not being routed down the wireguard tunnel at all.

My wireguard config states to route 0.0.0.0/0 (all) traffic via the wireguard tunnel connected, so I'm not sure why it's not doing that.

Any ideas?

[foss-johnny]

...

[keeka]

I also use a cloudflare A record for my vpn client conf.
wg client conf specifies a local DNS server (pihole) and search domain (local.lan).
Opnsense's unbound is the upstream server for the pihole.
Wireguard interface wg0 is assigned to a specific interface (VPN) and rules on there permit access to the local pihole instance amongst other things.

When using openvpn, prior to wireguard, I tried various configs to get the smoothest roaming experience. I ended up with port forwarding to localhost (for both WAN and LAN) because I found the openvpn server not reliably listening on all interfaces. When I switched to wireguard, all I did was modify port aliases and of course configure wireguard.

I tried wireguard for first time very recently, after using openvpn for a long time. After setting up the FQ Codel scheduler as per the buffer bloat recipe, I began to see increased openvpn warnings re out of sequence packets. So, thought it a good time to try Wireguard.

I am not sure why you see intermittent connectivity when connected to WG on the LAN. I notice the WG android client log is not very detailed.

[foss-johnny]

I also use a cloudflare A record for my vpn client conf.
wg client conf specifies a local DNS server (pihole) and search domain (local.lan).
Opnsense's unbound is the upstream server for the pihole.
Wireguard interface wg0 is assigned to a specific interface (VPN) and rules on there permit access to the local pihole instance amongst other things.

When using openvpn, prior to wireguard, I tried various configs to get the smoothest roaming experience. I ended up with port forwarding to localhost (for both WAN and LAN) because I found the openvpn server not reliably listening on all interfaces. When I switched to wireguard, all I did was modify port aliases and of course configure wireguard.

I tried wireguard for first time very recently, after using openvpn for a long time. After setting up the FQ Codel scheduler as per the buffer bloat recipe, I began to see increased openvpn warnings re out of sequence packets. So, thought it a good time to try Wireguard.

I am not sure why you see intermittent connectivity when connected to WG on the LAN. I notice the WG android client log is not very detailed.

Thanks for the further info and insights on your config.

Do you have any issues when roaming from 5g to wifi, or does it seamlessly roam without a problem?

I'm finding when switching from 5g to wifi, I need to turn off/on wifi and wg off/on and then it works correctly again. As if the routing needs to be reset.

Do you ever need to do this?

[keeka]

I'm finding when switching from 5g to wifi, I need to turn off/on wifi and wg off/on and then it works correctly again. As if the routing needs to be reset.

Do you ever need to do this?

Other than brief delays when the phone switches from wifi to mobile, no. I never need to restart android client or toggle wifi/mobile. Having said that I am not a heavy mobile user. But the only times I experience VPN connection issues is if I am out and lose mobile signal.