Wireguard in opnsense

[mimugmail]

Ok, I haven't tested such a setup, it might be possible.

At first, you need to add your tunnel network from wginbound to the outbound nat rule source network, as it includes only LAN and this doesn't match your wginbound.

Second, you should have a WGINBOUND rules tab, there you need a rules for allowing wginbound net to LAN (without gateway) and then a second rules with source wginbound and destination any, accept, gateway the mullvad gateway.

This should be enough.

[JDtheHutt]

Thank you for replying.  It confuses me further though, as I believe I had those rules previously, without it working as expected.  I've run a similar OpenVPN setup for years with many clients connecting and going out via a gateway to a VPN provider in the same way, and no issues.

I still have no connectivity from my external devices, including my mobile Android device, via WGINBOUND to my LAN or to the internet via MULLVAD.  However, I have noticed that there is no handshake occurring between my mobile and WG on OPNsense.

If I deactivate my Mullvad WG and revert to my WAN then I can connect my mobile via WGINBOUND and I see a handshake, and can access my LAN and the internet through my WAN, with effectively the same rules used, just going back out via my raw WAN rather than Mullvad.

With the Mullvad rules active, if I disable the open port rule in IDNET then in my firewall logs there is a block event for an IP linked to my mobile provider with a destination of the port.  If I enable the rule again, I see that traffic now passing and I would expect to see WG report a handshake but there is nothing.  If I activate my wifi and connect locally, I immediately see a handshake and I can access LAN and Mullvad.

Here are my amended areas as you advised, please tell me if I have entered anything incorrectly here, or if I am missing something.  Thank you again.

Firewall: NAT


Firewall: WGINBOUND


Firewall: IDNET (WAN)

[mimugmail]

Perhaps we can check with teamviewer when we hit a good timezone .. just come to irc and ping me

[JDtheHutt]

Perhaps we can check with teamviewer when we hit a good timezone .. just come to irc and ping me

Thank you, though no idea when that would be due to my shifts and delightful children .

I have been digging online and testing further though, and I think I may have a glimpse as to what it could be, though I'm still really unsure.

The Mullvad WG peer settings include 0.0.0.0/0, which I believe tells the system to push all packets through this WG tunnel. And I think this may be what is affecting the system, as it is forcing all other WG traffic down this tunnel regardless of my other firewall rules set in OPNsense. I'm seeing a lot of others having the same issue when running multiple interfaces with differing traffic routes. It seems that when one of the interfaces has 0.0.0.0/0 set, it pulls all the traffic into it. The solutions seem to usually be to exempt specific networks going into that WG tunnel, but I am struggling to understand the details for this. And I may have completely misunderstood it anyway.

Does this sound as if it could be a possible reason for the issue? I did test by setting my Mullvad peer address to something other than 0.0.0.0/0 and my Mullvad connection dropped, but my inbound connection immediately connected.

[mimugmail]

Ah, ok. That was the reason why I added a "Don't pull routes" checkbox
But then adding a gateway is much more tricky .. have to search the snipped I wrote.

Just try this feature and come back, until then I'll have certainly found the guide.

[JDtheHutt]

Ah, ok. That was the reason why I added a "Don't pull routes" checkbox
But then adding a gateway is much more tricky .. have to search the snipped I wrote.

Just try this feature and come back, until then I'll have certainly found the guide.

I've ticked the "Disable Routes" box in the WG Local setting for WGINBOUND and MULLVAD, which has resulted in a handshake for WGINBOUND and no handshake for MULLVAD but also no LAN or WAN connectivity for my mobile devices connecting via WGINBOUND.  I have tried stripping my rules out entirely now and going back to a basic, as default setup, then building from scratch my WGINBOUND setup with the "Disable Routes" box ticked, and I still achieve a handshake but still no connectivity to LAN to WAN.

If you do have a guide that could help, I would really appreciate it.  Though I'm beginning to think that I lack the capability to understand this at this time.  I've had numerous and more complex OpenVPN connections similar to this running successfully for years, but for WG I'm at a bit of a loss, feel like a chimp poking around till something clicks.

[mimugmail]

Only Dont pull routes for mullvad. And then you need to add a Gateway for mullvad, then in LAN Tab an Accept Rule with Destination whatever to mullvad VPN, then perhaps a Rule on wginbound Rule Tab with wginbound net to LAN Just Accept and a second Rule wginbound to any and again Gateway mullvad.

For debug you have to tcpdump on all Interfaces to check ifcpacket leaves (correct Routing) and/or gets natted

[JDtheHutt]

Thanks for replying again. I've certainly got further. Everything is tweaked as per your comments and I have a WGINBOUND connection again. However no WG handshake with Mullvad. At least the rules are correctly stopping outbound traffic escaping via my raw WAN now, as that was an issue before too.

There must be some rule or option I'm missing which is needed for the WG handshake and which is automatically added by WG if I don't tick "disable routes". That seems to be the last piece I need and I'm hoping you know what it is please.

After all this, I think maybe I should write this up to help others doing the same, unless that's something you're already doing?

[JDtheHutt]

Still working at this, and I believe my issues have been due to a lack of knowledge about some fundamental networking.  In the past, my mashed together setups seem to have worked due to luck and some auto rules from pfSense/OPNsense.  However, in this setup, the auto rules for the routes set by the Wireguard config do not allow them to work in conjunction.  I can have one running but not the other.  And disabling auto rules for both stops both working.

I was struggling a bit to work out what the auto rules it sets are.  It would be great if auto-defined rules could show up in the appropriate GUI window and list where they were set from.  Anyway, I realised that my NAT outbound rules likely needed to be set to allow each of the parts to be able to talk i.e. LAN, WGINBOUND, MULLVAD and WAN.  So I have tweaked my NAT outbound as below, and disabled the autorules in both WGINBOUND and MULLVAD WG configs.  This immediately saw handshakes for both, which is the first time that has happened, though still no connectivity from my mobile to my LAN or MULLVAD, and my LAN also lost MULLVAD connectivity.  I cannot ping either, so not just DNS.

Are these NAT rules correct for what I want?  Is there more NAT needed?  Do I need port forwards for any of this?  And do my firewall rules need changing, or should they still be fine as posted earlier?  Any further advice would be very appreciated, I think I'm close to sorting this.

[JDtheHutt]

I'm going through my routing tables to compare how they look when auto routes set to not set.  At least I'm slowly learning some new concepts, probably should have done this a while ago.  Still not sure what I need to set though, but I'm assuming it'll be one of those moments where I finally work it out and realise it's stupidly simple, if you know what it is in the first place.


EDIT: These are the additional routes I see if I have the Mullvad WG active and able to set routes itself.  I assume these are what I need to add to make this work manually, but I am not sure how I go about doing that or if and how they need to be tweaked to work alongside my WGINBOUND setup.

The third one's destination is the public exit IP for Mullvad's GB3 WG exit, and the gateway is my ISP's.

Code: [Select]

Proto       Destination      Gateway       Flags   Use       MTU      Netif       Netif (name)
ipv4     0.0.0.0/1      wg1    US      150      1412     wg1   MULLVADWG
ipv4     128.0.0.0/1      wg1    US      181      1412     wg1   MULLVADWG
ipv4     185.16.85.130    212.69.63.36  UGHS    141299    1492     pppoe0      IDNET


[JDtheHutt]

Actually, I think this is more a routing/rules issue than with WG. Is there a way to split my posts out into a new topic? Or can only an admin do that?

[JDtheHutt]

I finally managed it, but it is a Frankenstein's monster of a solution so far.

From my frantic reading and attempting to learn some more networking, I was under the impression that with static routes a packet would be sent to the one that most appropriately matches i.e. a packet for 8.8.8.8 would go through a route set to that IP, even if a route for 0.0.0.0/0 was before it.  I've seen people confirming this with their own setups.  But this has been confusing me, as my NAT and firewall rules appear correct and they work for both my outbound VPN provider and inbound remote client VPN just fine, I just can't get both tunnels up at the same time.  If I change my outbound VPN to something other than 0.0.0.0/0 then my inbound VPN immediately establishes, but I lose my outbound VPN.  I tried running without WG at all, then with each independently, then with both together, and examining the defined routes each time, especially the static routes WG established (they get set automatically in OPNsense).  They seemed correct and, as each worked independently, I assumed should still work together.  So I disabled OPNsense's auto-static route option and tried defining them manually myself.  This still didn't work.  I'm not sure if there's a difference in how WG interacts with static routes which is causing this.

So I went to an extreme solution.  I copied every assigned /8 IPv4 address block from Wikipedia at https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_address_blocks into a spreadsheet.  I first removed private address blocks 10.0.0.0/8, 172.0.0.0/8, 192.0.0.0/8, and my VPN provider endpoint which falls under 185.0.0.0/8, and added 127.0.0.0/8.  I sorted them in order then wrote a formula to concatenate them all and separate with a comma.  I copied this into the OPNsense config field so that they were recognised as individual address blocks.  Saved and started up both WG tunnels.  And both immediately established, with my LAN having internet access through the VPN provider as well as my mobile able to access my LAN and then from there the internet through the VPN provider.

It's a hell of a bodge.  My states table looks like a crumbling ruin with 240 separate entries to cover nearly every /8 address block and it's not the quickest of systems, as I assume it's now having to parse all of that before each packet is routed, but it works.  I don't know why the earlier attempts didn't.  Perhaps it is an issue with OPNsense or its WG implementation in how the static routes and associated packets operate, or there's still some config nuance I haven't grasped, or something else entirely.  I'm going to continue testing and try to work out what exactly is going on, as I don't feel this is a viable long-term solution and I'm hoping that at least knowing it can work will help something click in my head to get a proper config together.  I've seen mention of fwmark and I think that may be what I am missing so far: a means to exclude certain packets from going into a WG tunnel, as that seems a cleaner way to do this rather than have to strictly define most of the internet to go through the tunnel.  There's no exclusion option in the OPNsense GUI, so I'll have to try and see if this can be made to work manually on the CLI.

If I get it working without it being the abomination above, I'll let you know and get some screengrabs along with a clearer description.

[guest15389]

Is there any more help you need in testing the WireGuard plugin to move it out of development?

I've just converted over my Mac to use Wireguard since the app was launched just recently.

I installed the plugin and basically setup the RoadWarrior setup and it's working well.

The directions for the "road warrior" setup seemed a little confusing but if I can get some time, I can do a PR to help clean that up.

Great work!

[mimugmail]

As long as this is not fixed, no chance:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233955

[guest15389]

Hmm, that's an odd bug. I haven't noticed it yet on my setup as I've been running it, but I really don't play around with it too much other than letting it run at this point.