Policy based routing with WireGuard plugin

[Rainmaker]

NOTE: This thread started as a question, but I solved it myself. Please treat it as a mini-guide on how to get WireGuard working with AzireVPN (or Mullvad etc) while still allowing yourself to access servers on your LAN from outside the network, using poilcy based routing. I hope my half-week of frustration, lack of sleep and grey hairs helps at least one other person.

Thanks to the guide of a fellow forum member on his website Routerperformance, I have the experimental WireGuard plugin installed and working on OPNsense v18.7.

The only issue is, I have a LAN node (TiVO) that must be connected directly to my cable ISP to work properly. I also have another local client (NAS) that hosts Plex, SABnzbd and some other stuff that I'd prefer to either similarly route directly to the ISP WAN, or else have a way to still forward ports using DNAT from WAN IP > NAS along the LAN (i.e. from 192.168.1.1 at the firewall to 192.168.1.5 at the NAS and back).

I did experiment with policy based routing, by assigning the wg0 link to an interface in OPNsense and then assigning a gateway. I changed the LAN firewall rules to:

1. $(Alias for bypass LAN IPs) - any - any - WAN_GW
2. 192.168.0.0/24 - any - any - AzireVPN_GW #Supposedly to route all other LAN traffic via VPN as they wouldn't have matched the first rule

The LAN clients just don't connect now. So I also tried changing outbound NAT (in manual mode) to:

1. WAN_DHCP - $(Alias for bypass LAN IPs) - any - any - WAN_ADDRESS
2. AzireVPN - 192.168.0.0/24 - any - any - AZIRE_TUNNEL_ADDRESS

And still nothing. I'm a bit stuck. I'll be honest I gave up on this once already, and wiped my machine and installed Arch Linux base. I added dnscrypt-proxy for dns, dhcpd for DHCP server, wireguard-tools for VPN and Shorewall to control the netfilter firewall. I assigned zones to WAN, LAN and WireGuard and away I went. I'm typing from it now and it's great... except I similarly am bumping into issues with policy based routing.

I have to choose at present, regardless of whether I'm using Arch or OPNsense, to either (a) have AzireVPN on the router - my preferred option - but have some restrictions on my TiVO and have my NAS be inaccessible from outside the LAN; or (b) have the router without VPN and just connect every individual LAN node to WireGuard, so at least my TiVO and NAS work properly. That's not actually ideal, as some local nodes aren't powerful enough to run a VPN locally without running out of steam (I have a relatively fast linespeed from my ISP).

I assume the reason it's breaking under both OPNsense and Linux is that the wg-quick tools add their own routing which completely cuts off the firewall from the ISP/WAN interface. I remember a few years ago I had IPSec tunnels up instead, and could just selectively route without issues. Incoming WAN packets to my servers (eg Plex on NAS) were hitting the firewall, and getting routed locally from $FW_IP to $NAS_LAN_IP and getting answered that way. So at the time I had the best of both worlds - all LAN clients were safe behind a VPN for all their activities, but I could still access my personal servers when away from home using my ISP's WAN IP or domain name.

Can someone help me work this out? I did try checking the 'disable routes' box on the WireGuard server page, expecting to be able to manually assign routes using the gateway and NAT rules, but the tunnel just never comes up.    All help gratefully appreciated! Thanks in advance.

[Rainmaker]

By the way, my SFF edge router/firewall box only has two onboard Intel NICs. It's literally smaller than a piece of paper (a little larger than a PC Engines APU, for those in the know), so it's not possible to add any more NICs. I do realise the absolute easiest solution here is to have maybe 4 NICs instead of 2. That way I could have WAN, LAN1, DMZ1 and DMZ2 and hook up my 'normal' LAN nodes via switch to LAN1 and then plug in my TiVO and NAS to DMZ1 and DMZ2. That would make it trivial to add zones in Shorewall (Linux) to bypass the VPN routing on LAN1, and I'm certain it would be equally easy to do on OPNsense.

Unfortunately I spent a lot of money on this little PC (relatively speaking) only this year, so if at all possible I don't want to have to buy more hardware! A software/policy resolution to this issue would be preferable by far.

[Rainmaker]

Please forgive me talking to myself here, but I think I actually cracked it. I set up a couple of virtual machines in VMWare Workstation 15. The first was OPNsense 18.7 and the second was a generic Linux guest to use as a test 'LAN' client.

My setup actually goes like this:

ISP WAN > Arch Linux router/firewall > Desktop PC with VMWare > OPNsense

Quite a few ducks to get in a row, as the saying goes! I opened a random made up port (32401) on my 'real' Arch firewall and forwarded it to the LAN IP of the OPNsense VM (its' WAN IP as it were).

I have two virtual NICs on the OPNsense virtual machine. The first is bridged to the host's ethernet NIC and is "WAN" for the OPNsense machine. The second is tied to a LAN segment called 'Router'. The random Linux VM's NIC is attached to the same 'Router' LAN segment (aka I turned OPNsense into a virtual router and connected a Linux install to it as a pretend LAN client).

OPNsense was then connected to AzireVPN using the WireGuard plugin. Outbound NAT was put into manual mode and a single entry was made to send all LAN net traffic (172.16.0.0/24) via the AzireVPN gateway.

On the LAN firewall I added two rules. The top rule (i.e. matched first) was to send traffic from 172.16.0.150 (my Linux guest's LAN IP) and port 32401 via the WAN gateway. The second rule says to send all LAN traffic via AzireVPN's gateway.

Finally I added a port forward for 'from any to WAN address port 32401' and redirected it to 172.16.0.150. This then added the resulting rule to the WAN firewall.

We need a server. The Linux install had Transmission by default so I enabled Remote Access on that (i.e. its webUI) and set it to port 32401.

The moment of truth. I grabbed my mobile phone and turned off wifi. Once on 4G I tried to connect to mydomain.com:32401 and boom - Transmission's webUI! I went back to the Linux virtual machine and fired up a browser, and it works perfectly. Visiting browserleaks.com/ip or any similar site shows my IP is the AzireVPN one, and Azire's connection checker says I'm connected to their VPN. Success! So it seems I solved my own problem after two days of pulling out my hair across two different firewall operating systems!

I just have to wipe my SFF router box and install OPNsense again, and more importantly replicate this on a 'real' install with a great many forwards and rules. Should be easy enough, in essence I just need to install OPNsense, get WireGuard up and running, add all my port forwards as is usual for any OPNsense install, but then add secondary rules for those ports on the LAN firewall as well, to point them to WAN rather than Azire. A final (bottom/last) rule on the LAN firewall sends all LAN traffic to Azire (i.e. any traffic not matching a local server rule) and finally completes the set.

Wish me luck, and if it works I'll edit this post to mark it successful; so hopefully anyone searching for this topic in future can be helped by it.

[Rainmaker]

It works! \0/







Note to any readers: I didn't need to add more LAN rules on the 'real'/physical machine. Whether that was an artifact of the VM being double NATed or the fact I enabled all three types of Hairpin NAT (DNAT) this time, I'm unsure. But tonight, for the first time in three days, I'll sleep like a baby haha. I hope this helps someone. 

[Rainmaker]

Oh, and it would be remiss not to include the speed test:



For context, my ISP line speed without VPN is a maximum of 380 down. My router is a Kaby Lake Pentium 4560 2c4t @3.5GHz, with 4GB 2333MHz DDR4 and a 32GB mSATA SSD (600MB/sec read/write). During the speed test the maximum CPU usage was 20% with 1.1% interrupt. Considering that's in user space on BSD and not kernel space, that's really good going!

Unfortunately I'm now hitting issues with the plugin itself, crashing a lot (and every reboot). I'm troubleshooting this but I do know that's an issue with the (beta) plugin rather than the config. Hopefully a bugfix/patch is issued soon. Either way, we know it works!

[mimugmail]

Unfortunately I'm now hitting issues with the plugin itself, crashing a lot (and every reboot). I'm troubleshooting this but I do know that's an issue with the (beta) plugin rather than the config. Hopefully a bugfix/patch is issued soon. Either way, we know it works!

No, the plugin is stable, the freebsd port itself has problems. Or better the freebsd kernel when you installed OPNsense on UFS. It causes stack traces (so, reboots). For some it works, for some not .. but it's not in our hand anymore.

[Rainmaker]

Unfortunately I'm now hitting issues with the plugin itself, crashing a lot (and every reboot). I'm troubleshooting this but I do know that's an issue with the (beta) plugin rather than the config. Hopefully a bugfix/patch is issued soon. Either way, we know it works!

No, the plugin is stable, the freebsd port itself has problems. Or better the freebsd kernel when you installed OPNsense on UFS. It causes stack traces (so, reboots). For some it works, for some not .. but it's not in our hand anymore.

Ah, apologies. When I said 'the plugin' I did of course just generically mean 'something WireGuard related that wasn't broken before I added it'. Unfortunately since my success my connection from LAN clients suddenly ground to a halt. The OPNsense dashboard shows plenty of activity (traffic graph, firewall log) between LAN, WAN and AZIRE but alas nothing turns up on the actual LAN machines. Ping is possible on them, but web pages time out and iperf3 to a remote host connects (as with ping) but traffic throughput is 0.00. Weird.

I guess I'm back to having 10 or so tunnels on 10 or so local devices again until (hopefully) something gets resolved. I'd spent so long trying to get this working on OPNsense and Arch I've literally pulled an all-nighter again. C'est la vie.

PS: I never had any forced reboots, just that every time I do reboot I get a crash report about the wireguard plugin's interface php file (or something, I submitted them anyway). Since they started my connection disappeared.

[JDtheHutt]

EDIT: my tired brain just realised I misread your post so please ignore as I can't even work out how to delete this. My issue is getting mobile clients to connect via WG while also having a WG VPN provider connection to Mullvad.

Hi Rainmaker. I believe I am trying to do the same as you did here, only with Mullvad instead. Mimugmail has helped me get most of the way there as I can handshake from my mobile to OPNsense and connect to my LAN via WG, but I can't get Mullvad to handshake. I think I'm close and that what you have worked out is the last piece.

My brain is utterly fried though and I can't work it out from your post. Is there any chance you could screenshot your specific rules that resolved this and post them so that I can get this working? I would really appreciate it.

[mimugmail]

This was a note to myself in IRC:

wireguard is tun, p2p .. so your interface is inet 10.30.8.124 --> 10.30.8.124  netmask 0xffffe000 .. if you want to policy route via pf, you have to create a gateway, with IP dynamic. then you have to create a IP gateway with a IP withing the range, like 10.30.8.1 .. and then a route, destination 10.30.8.1 via the first gateway. then you can do policy routing when setting gateway2 in rules. ¯\_(ツ)_/¯