Messages

Hi Franco. Yeah I saw the patches from Jason on the bug report. He's a good guy. I've been on the WireGuard mailing list since forever, it's how I spotted the plugin had gone stable on here. Thanks again. :)

Ah so it was (at least) two issues in the end. Nice to see it so well tended and fixed. Thanks.

I regrettably had to move away from OPNsense to better use WireGuard on my router. At that time, the great work being done on (what was then named) os-wireguard-devel was being hindered by an upstream bug in FreeBSD. This was causing kernel panics and crashes when running WireGuard on UFS systems like OPNsense. That's the best of my recollection, anyway.

In the meantime, I wiped the Dell Optiplex 7010 (i7 3700, 8GB RAM, Intel Pro 1000PT server NIC) that lives at the edge of my network. Basically I 'made' my own router from scratch using Arch Linux, dnscrypt-proxy, WireGuard, Shorewall and so on. This has been working OK but it's a bit 'hacky' and cobbled together.

I was delighted to see that OPNsense now has a stable release os-wireguard, and from limited testing in a VM it seems OK (it's hard to properly test OPNsense between VMs due to my home network setup). Can anyone please confirm that the above bug has been fixed upstream, and that I should be safe (as anyone can be) to reinstall OPNsense on my router and set up WireGuard on there for one of my LAN subnets? I am to keep my network as follows:

WAN (cable modem, DHCP)
LAN1 > ProSafe switch > (trusted, local devices etc)
LAN2 > ProSafe switch > (DMZ, servers and NAS, WiFi, IoT)
wg0 (routing devices from LAN1)

Thanks in advance.

Unfortunately I'm now hitting issues with the plugin itself, crashing a lot (and every reboot). I'm troubleshooting this but I do know that's an issue with the (beta) plugin rather than the config. Hopefully a bugfix/patch is issued soon. Either way, we know it works!

No, the plugin is stable, the freebsd port itself has problems. Or better the freebsd kernel when you installed OPNsense on UFS. It causes stack traces (so, reboots). For some it works, for some not .. but it's not in our hand anymore.

Ah, apologies. When I said 'the plugin' I did of course just generically mean 'something WireGuard related that wasn't broken before I added it'. Unfortunately since my success my connection from LAN clients suddenly ground to a halt. The OPNsense dashboard shows plenty of activity (traffic graph, firewall log) between LAN, WAN and AZIRE but alas nothing turns up on the actual LAN machines. Ping is possible on them, but web pages time out and iperf3 to a remote host connects (as with ping) but traffic throughput is 0.00. Weird.

I guess I'm back to having 10 or so tunnels on 10 or so local devices again until (hopefully) something gets resolved. I'd spent so long trying to get this working on OPNsense and Arch I've literally pulled an all-nighter again. C'est la vie.

PS: I never had any forced reboots, just that every time I do reboot I get a crash report about the wireguard plugin's interface php file (or something, I submitted them anyway). Since they started my connection disappeared.

Oh, and it would be remiss not to include the speed test:



For context, my ISP line speed without VPN is a maximum of 380 down. My router is a Kaby Lake Pentium 4560 2c4t @3.5GHz, with 4GB 2333MHz DDR4 and a 32GB mSATA SSD (600MB/sec read/write). During the speed test the maximum CPU usage was 20% with 1.1% interrupt. Considering that's in user space on BSD and not kernel space, that's really good going!

Unfortunately I'm now hitting issues with the plugin itself, crashing a lot (and every reboot). I'm troubleshooting this but I do know that's an issue with the (beta) plugin rather than the config. Hopefully a bugfix/patch is issued soon. Either way, we know it works!

It works! \0/







Note to any readers: I didn't need to add more LAN rules on the 'real'/physical machine. Whether that was an artifact of the VM being double NATed or the fact I enabled all three types of Hairpin NAT (DNAT) this time, I'm unsure. But tonight, for the first time in three days, I'll sleep like a baby haha. I hope this helps someone.  8)

[but]

Please forgive me talking to myself here, but I think I actually cracked it. I set up a couple of virtual machines in VMWare Workstation 15. The first was OPNsense 18.7 and the second was a generic Linux guest to use as a test 'LAN' client.

My setup actually goes like this:

ISP WAN > Arch Linux router/firewall > Desktop PC with VMWare > OPNsense

Quite a few ducks to get in a row, as the saying goes! I opened a random made up port (32401) on my 'real' Arch firewall and forwarded it to the LAN IP of the OPNsense VM (its' WAN IP as it were).

I have two virtual NICs on the OPNsense virtual machine. The first is bridged to the host's ethernet NIC and is "WAN" for the OPNsense machine. The second is tied to a LAN segment called 'Router'. The random Linux VM's NIC is attached to the same 'Router' LAN segment (aka I turned OPNsense into a virtual router and connected a Linux install to it as a pretend LAN client).

OPNsense was then connected to AzireVPN using the WireGuard plugin. Outbound NAT was put into manual mode and a single entry was made to send all LAN net traffic (172.16.0.0/24) via the AzireVPN gateway.

On the LAN firewall I added two rules. The top rule (i.e. matched first) was to send traffic from 172.16.0.150 (my Linux guest's LAN IP) and port 32401 via the WAN gateway. The second rule says to send all LAN traffic via AzireVPN's gateway.

Finally I added a port forward for 'from any to WAN address port 32401' and redirected it to 172.16.0.150. This then added the resulting rule to the WAN firewall.

We need a server. The Linux install had Transmission by default so I enabled Remote Access on that (i.e. its webUI) and set it to port 32401.

The moment of truth. I grabbed my mobile phone and turned off wifi. Once on 4G I tried to connect to mydomain.com:32401 and boom - Transmission's webUI! I went back to the Linux virtual machine and fired up a browser, and it works perfectly. Visiting browserleaks.com/ip or any similar site shows my IP is the AzireVPN one, and Azire's connection checker says I'm connected to their VPN. Success! So it seems I solved my own problem after two days of pulling out my hair across two different firewall operating systems!

I just have to wipe my SFF router box and install OPNsense again, and more importantly replicate this on a 'real' install with a great many forwards and rules. Should be easy enough, in essence I just need to install OPNsense, get WireGuard up and running, add all my port forwards as is usual for any OPNsense install, but then add secondary rules for those ports on the LAN firewall as well, to point them to WAN rather than Azire. A final (bottom/last) rule on the LAN firewall sends all LAN traffic to Azire (i.e. any traffic not matching a local server rule) and finally completes the set.

Wish me luck, and if it works I'll edit this post to mark it successful; so hopefully anyone searching for this topic in future can be helped by it.

By the way, my SFF edge router/firewall box only has two onboard Intel NICs. It's literally smaller than a piece of paper (a little larger than a PC Engines APU, for those in the know), so it's not possible to add any more NICs. I do realise the absolute easiest solution here is to have maybe 4 NICs instead of 2. That way I could have WAN, LAN1, DMZ1 and DMZ2 and hook up my 'normal' LAN nodes via switch to LAN1 and then plug in my TiVO and NAS to DMZ1 and DMZ2. That would make it trivial to add zones in Shorewall (Linux) to bypass the VPN routing on LAN1, and I'm certain it would be equally easy to do on OPNsense.

Unfortunately I spent a lot of money on this little PC (relatively speaking) only this year, so if at all possible I don't want to have to buy more hardware! A software/policy resolution to this issue would be preferable by far.

[NOTE: This thread started as a question, but I solved it myself. Please treat it as a mini-guide on how to get WireGuard working with AzireVPN (or Mullvad etc) while still allowing yourself to access servers on your LAN from outside the network, using poilcy based routing. I hope my half-week of frustration, lack of sleep and grey hairs helps at least one other person.]

NOTE: This thread started as a question, but I solved it myself. Please treat it as a mini-guide on how to get WireGuard working with AzireVPN (or Mullvad etc) while still allowing yourself to access servers on your LAN from outside the network, using poilcy based routing. I hope my half-week of frustration, lack of sleep and grey hairs helps at least one other person.

Thanks to the guide of a fellow forum member on his website Routerperformance, I have the experimental WireGuard plugin installed and working on OPNsense v18.7.

The only issue is, I have a LAN node (TiVO) that must be connected directly to my cable ISP to work properly. I also have another local client (NAS) that hosts Plex, SABnzbd and some other stuff that I'd prefer to either similarly route directly to the ISP WAN, or else have a way to still forward ports using DNAT from WAN IP > NAS along the LAN (i.e. from 192.168.1.1 at the firewall to 192.168.1.5 at the NAS and back).

I did experiment with policy based routing, by assigning the wg0 link to an interface in OPNsense and then assigning a gateway. I changed the LAN firewall rules to:

1. $(Alias for bypass LAN IPs) - any - any - WAN_GW
2. 192.168.0.0/24 - any - any - AzireVPN_GW #Supposedly to route all other LAN traffic via VPN as they wouldn't have matched the first rule

The LAN clients just don't connect now. So I also tried changing outbound NAT (in manual mode) to:

1. WAN_DHCP - $(Alias for bypass LAN IPs) - any - any - WAN_ADDRESS
2. AzireVPN - 192.168.0.0/24 - any - any - AZIRE_TUNNEL_ADDRESS

And still nothing. I'm a bit stuck. I'll be honest I gave up on this once already, and wiped my machine and installed Arch Linux base. I added dnscrypt-proxy for dns, dhcpd for DHCP server, wireguard-tools for VPN and Shorewall to control the netfilter firewall. I assigned zones to WAN, LAN and WireGuard and away I went. I'm typing from it now and it's great... except I similarly am bumping into issues with policy based routing.

I have to choose at present, regardless of whether I'm using Arch or OPNsense, to either (a) have AzireVPN on the router - my preferred option - but have some restrictions on my TiVO and have my NAS be inaccessible from outside the LAN; or (b) have the router without VPN and just connect every individual LAN node to WireGuard, so at least my TiVO and NAS work properly. That's not actually ideal, as some local nodes aren't powerful enough to run a VPN locally without running out of steam (I have a relatively fast linespeed from my ISP).

I assume the reason it's breaking under both OPNsense and Linux is that the wg-quick tools add their own routing which completely cuts off the firewall from the ISP/WAN interface. I remember a few years ago I had IPSec tunnels up instead, and could just selectively route without issues. Incoming WAN packets to my servers (eg Plex on NAS) were hitting the firewall, and getting routed locally from $FW_IP to $NAS_LAN_IP and getting answered that way. So at the time I had the best of both worlds - all LAN clients were safe behind a VPN for all their activities, but I could still access my personal servers when away from home using my ISP's WAN IP or domain name.

Can someone help me work this out? I did try checking the 'disable routes' box on the WireGuard server page, expecting to be able to manually assign routes using the gateway and NAT rules, but the tunnel just never comes up.  :(  All help gratefully appreciated! Thanks in advance.

Thanks, Franco. That fixed it. ;)

I have setup OPNsense to run SSH over a non-standard port. Login group is set to wheel and admins. Root login is disabled (box unchecked). However, when I try to SSH into the box as 'user' (who is a member of admins), I am prompted for the password. The password is accepted and the OPNsense logo appears, but followed immediately by a message that I 'must be root to login'. As I said, 'permit root user login' is unchecked, and the root user account is disabled in System > Access > Users!

The only way around this is to enable the root user, and log in via SSH using root. My 'user' is a member of admins, with permissions inherited from admins. What am I missing? It's obviously much less secure to enable the root account for SSH than to log in as 'user' and use sudo.

Hi again,

Just to let you know I followed the guide and this works perfectly. Thanks! My one bugbear about unbound is the fact it can't reuse TLS sessions if you enable DoT, making lookups slower. Now I have speedy lookups without losing the encryption. Perfect! Just a couple of little pointers for v2 of the guide (if you ever release it):

1) You missed the first forward slash off the paths, eg 'etc/rc.conf' instead of '/etc/rc.conf'.
2) The command to enable dnscrypt-proxy was given as 'sudo service dnscrypt-proxy enable' but should have been 'sudo service dnscrypt-proxy enabled' (note the 'd' on the end).

Other than that, sterling work. Thank you so much!

That looks like it's going to be a great guide. Thanks for taking the time! Unfortunately most of the text boxes are blank (with command line output), though the images all seem to have embedded fine. Can you take another look and re-upload? I'm interested in reading it as I rely on forwarding to DNS over TLS atm, which isn't ideal using Unbound. As you probably know, Unbound can't reuse TLS connections and has to make new ones for every query, slowing down DNS resolution. Using dnscrypt-proxy would fix this nicely.

Hi pylox,

Having read the relevant man pages it seems I was indeed grossly misinformed. I was told to add those lines when I first started using pfSense (and *BSD), as I was using various Intel Pro and I-series ethernet NICs. I don't run wifi on my gateway (I use Unifi APs) so I was obviously given duff information.

My apologies for repeating it here, and thanks for the lesson.

Rainmaker,

I think I'm not communicating well. I'm not saying that FreeBSD has poor network performance. I'm saying that with the particular piece of hardware I happen to own FreeBSD has roughly half the throughput of linux and is struggles to use it efficiently. The FreeBSD development thread listed earlier suggests that it's not the SMP performance of the pf that's the issue, but something to do with some oddness in the embedded intel NIC.

But, most of these benchmarks are almost two years old. I'm wondering if at this point the problem with this particular hardware might be fixed and if people might be able to get similar performance under FreeBSD with tweaks.

Ah, you (respectfully) are a lot more knowledgeable than I catered for in my response. Apologies, it's difficult to pitch your responses on the Internet; especially when you and the other people don't know each other yet (as I'm sure you know).

Yes, FreeBSD's pf is indeed much more SMP capable. Last week I took both OpenBSD and FreeBSD installs and 'made' routers out of them, before comparing them side-by-side. Even on an 8700k at 5GHz per core OpenBSD was less performant than FreeBSD. However there are many other factors, as we both touched upon in previous posts.

NIC queues are one factor, as you state. I'm not sure if OPNsense utilises multiple queues (i.e. per core) or whether it just uses one. For your APU specifically, did you accept the Intel proprietary licence? I can't quite recall whether the APU2 uses igb drivers? I don't even know if that applies to OPNsense, but I know on pfSense I was advised to create the following for an APU2:

Code Select Expand

/boot/loader.conf.local
 
legal.intel_ipw.license_ack=1
legal.intel_iwi.license_ack=1

Then reboot. This apparently 'unlocks' some extra functionality in the NIC, which may improve your throughput. If you're running off an SSD don't forget to enable TRIM.