Messages

I finally managed it, but it is a Frankenstein's monster of a solution so far.

From my frantic reading and attempting to learn some more networking, I was under the impression that with static routes a packet would be sent to the one that most appropriately matches i.e. a packet for 8.8.8.8 would go through a route set to that IP, even if a route for 0.0.0.0/0 was before it.  I've seen people confirming this with their own setups.  But this has been confusing me, as my NAT and firewall rules appear correct and they work for both my outbound VPN provider and inbound remote client VPN just fine, I just can't get both tunnels up at the same time.  If I change my outbound VPN to something other than 0.0.0.0/0 then my inbound VPN immediately establishes, but I lose my outbound VPN.  I tried running without WG at all, then with each independently, then with both together, and examining the defined routes each time, especially the static routes WG established (they get set automatically in OPNsense).  They seemed correct and, as each worked independently, I assumed should still work together.  So I disabled OPNsense's auto-static route option and tried defining them manually myself.  This still didn't work.  I'm not sure if there's a difference in how WG interacts with static routes which is causing this.

So I went to an extreme solution.  I copied every assigned /8 IPv4 address block from Wikipedia at https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_address_blocks into a spreadsheet.  I first removed private address blocks 10.0.0.0/8, 172.0.0.0/8, 192.0.0.0/8, and my VPN provider endpoint which falls under 185.0.0.0/8, and added 127.0.0.0/8.  I sorted them in order then wrote a formula to concatenate them all and separate with a comma.  I copied this into the OPNsense config field so that they were recognised as individual address blocks.  Saved and started up both WG tunnels.  And both immediately established, with my LAN having internet access through the VPN provider as well as my mobile able to access my LAN and then from there the internet through the VPN provider.

It's a hell of a bodge.  My states table looks like a crumbling ruin with 240 separate entries to cover nearly every /8 address block and it's not the quickest of systems, as I assume it's now having to parse all of that before each packet is routed, but it works.  I don't know why the earlier attempts didn't.  Perhaps it is an issue with OPNsense or its WG implementation in how the static routes and associated packets operate, or there's still some config nuance I haven't grasped, or something else entirely.  I'm going to continue testing and try to work out what exactly is going on, as I don't feel this is a viable long-term solution and I'm hoping that at least knowing it can work will help something click in my head to get a proper config together.  I've seen mention of fwmark and I think that may be what I am missing so far: a means to exclude certain packets from going into a WG tunnel, as that seems a cleaner way to do this rather than have to strictly define most of the internet to go through the tunnel.  There's no exclusion option in the OPNsense GUI, so I'll have to try and see if this can be made to work manually on the CLI.

If I get it working without it being the abomination above, I'll let you know and get some screengrabs along with a clearer description.

Actually, I think this is more a routing/rules issue than with WG. Is there a way to split my posts out into a new topic? Or can only an admin do that?

I'm going through my routing tables to compare how they look when auto routes set to not set.  At least I'm slowly learning some new concepts, probably should have done this a while ago.  Still not sure what I need to set though, but I'm assuming it'll be one of those moments where I finally work it out and realise it's stupidly simple, if you know what it is in the first place.


EDIT: These are the additional routes I see if I have the Mullvad WG active and able to set routes itself.  I assume these are what I need to add to make this work manually, but I am not sure how I go about doing that or if and how they need to be tweaked to work alongside my WGINBOUND setup.

The third one's destination is the public exit IP for Mullvad's GB3 WG exit, and the gateway is my ISP's.

Code Select Expand


Proto       Destination      Gateway       Flags   Use       MTU      Netif       Netif (name)
ipv4     0.0.0.0/1      wg1    US      150      1412     wg1   MULLVADWG
ipv4     128.0.0.0/1      wg1    US      181      1412     wg1   MULLVADWG
ipv4     185.16.85.130    212.69.63.36  UGHS    141299    1492     pppoe0      IDNET


Still working at this, and I believe my issues have been due to a lack of knowledge about some fundamental networking.  In the past, my mashed together setups seem to have worked due to luck and some auto rules from pfSense/OPNsense.  However, in this setup, the auto rules for the routes set by the Wireguard config do not allow them to work in conjunction.  I can have one running but not the other.  And disabling auto rules for both stops both working.

I was struggling a bit to work out what the auto rules it sets are.  It would be great if auto-defined rules could show up in the appropriate GUI window and list where they were set from.  Anyway, I realised that my NAT outbound rules likely needed to be set to allow each of the parts to be able to talk i.e. LAN, WGINBOUND, MULLVAD and WAN.  So I have tweaked my NAT outbound as below, and disabled the autorules in both WGINBOUND and MULLVAD WG configs.  This immediately saw handshakes for both, which is the first time that has happened, though still no connectivity from my mobile to my LAN or MULLVAD, and my LAN also lost MULLVAD connectivity.  I cannot ping either, so not just DNS.

Are these NAT rules correct for what I want?  Is there more NAT needed?  Do I need port forwards for any of this?  And do my firewall rules need changing, or should they still be fine as posted earlier?  Any further advice would be very appreciated, I think I'm close to sorting this.

Thanks for replying again. I've certainly got further. Everything is tweaked as per your comments and I have a WGINBOUND connection again. However no WG handshake with Mullvad. At least the rules are correctly stopping outbound traffic escaping via my raw WAN now, as that was an issue before too.

There must be some rule or option I'm missing which is needed for the WG handshake and which is automatically added by WG if I don't tick "disable routes". That seems to be the last piece I need and I'm hoping you know what it is please.

After all this, I think maybe I should write this up to help others doing the same, unless that's something you're already doing?

EDIT: my tired brain just realised I misread your post so please ignore as I can't even work out how to delete this. My issue is getting mobile clients to connect via WG while also having a WG VPN provider connection to Mullvad.

Hi Rainmaker. I believe I am trying to do the same as you did here, only with Mullvad instead. Mimugmail has helped me get most of the way there as I can handshake from my mobile to OPNsense and connect to my LAN via WG, but I can't get Mullvad to handshake. I think I'm close and that what you have worked out is the last piece.

My brain is utterly fried though and I can't work it out from your post. Is there any chance you could screenshot your specific rules that resolved this and post them so that I can get this working? I would really appreciate it.

Ah, ok. That was the reason why I added a "Don't pull routes" checkbox :)
But then adding a gateway is much more tricky .. have to search the snipped I wrote.

Just try this feature and come back, until then I'll have certainly found the guide.

I've ticked the "Disable Routes" box in the WG Local setting for WGINBOUND and MULLVAD, which has resulted in a handshake for WGINBOUND and no handshake for MULLVAD but also no LAN or WAN connectivity for my mobile devices connecting via WGINBOUND.  I have tried stripping my rules out entirely now and going back to a basic, as default setup, then building from scratch my WGINBOUND setup with the "Disable Routes" box ticked, and I still achieve a handshake but still no connectivity to LAN to WAN.

If you do have a guide that could help, I would really appreciate it.  Though I'm beginning to think that I lack the capability to understand this at this time.  I've had numerous and more complex OpenVPN connections similar to this running successfully for years, but for WG I'm at a bit of a loss, feel like a chimp poking around till something clicks.

Perhaps we can check with teamviewer when we hit a good timezone .. just come to irc and ping me

Thank you, though no idea when that would be due to my shifts and delightful children .

I have been digging online and testing further though, and I think I may have a glimpse as to what it could be, though I'm still really unsure.

The Mullvad WG peer settings include 0.0.0.0/0, which I believe tells the system to push all packets through this WG tunnel. And I think this may be what is affecting the system, as it is forcing all other WG traffic down this tunnel regardless of my other firewall rules set in OPNsense. I'm seeing a lot of others having the same issue when running multiple interfaces with differing traffic routes. It seems that when one of the interfaces has 0.0.0.0/0 set, it pulls all the traffic into it. The solutions seem to usually be to exempt specific networks going into that WG tunnel, but I am struggling to understand the details for this. And I may have completely misunderstood it anyway.

Does this sound as if it could be a possible reason for the issue? I did test by setting my Mullvad peer address to something other than 0.0.0.0/0 and my Mullvad connection dropped, but my inbound connection immediately connected.

Thank you for replying.  It confuses me further though, as I believe I had those rules previously, without it working as expected.  I've run a similar OpenVPN setup for years with many clients connecting and going out via a gateway to a VPN provider in the same way, and no issues.

I still have no connectivity from my external devices, including my mobile Android device, via WGINBOUND to my LAN or to the internet via MULLVAD.  However, I have noticed that there is no handshake occurring between my mobile and WG on OPNsense.

If I deactivate my Mullvad WG and revert to my WAN then I can connect my mobile via WGINBOUND and I see a handshake, and can access my LAN and the internet through my WAN, with effectively the same rules used, just going back out via my raw WAN rather than Mullvad.

With the Mullvad rules active, if I disable the open port rule in IDNET then in my firewall logs there is a block event for an IP linked to my mobile provider with a destination of the port.  If I enable the rule again, I see that traffic now passing and I would expect to see WG report a handshake but there is nothing.  If I activate my wifi and connect locally, I immediately see a handshake and I can access LAN and Mullvad.

Here are my amended areas as you advised, please tell me if I have entered anything incorrectly here, or if I am missing something.  Thank you again.

Firewall: NAT


Firewall: WGINBOUND


Firewall: IDNET (WAN)

Can you rephrase in one sentence what you want to achieve and then give some facts (IP's) etc. and screenshot of rules, outbound nat, wireguard config, assigned interface and gateway

Desired setup: Wireguard connection between OPNsense and Mullvad through which all network traffic is transferred.  Additionally, separate Wireguard connections between OPNsense and some roaming external devices i.e. Android phones, laptops etc, which should be able to connect to my internal LAN devices and to the internet via my Mullvad Wireguard connection.

I possess a static IPv4 address with my ISP, which simplifies my personal WG connection between external devices and my home network as no need to mess with dynamic IP services.

Following guides including those at https://www.routerperformance.net/ and https://wiki.opnsense.org/manual/howtos.html I can get either my connection to Mullvad, or my personal connections from devices to my LAN, but I just can't seem to get both working at the same time.  At this time, I have deleted all Rules and NAT relating to my personal WG->LAN connections as it had become a mess of testing and wasn't really helping

Additionally, with my Mullvad connection, my traffic seems to correctly go via the gateway I have set up for the interface, but if I go to Firewall->Advanced and tick "Skip rules when gateway is down" then all traffic is blocked.  I'm not sure whether the gateway is working in the first place or if it is incorrectly believing the gateway is down. 

Also, any changes to my interfaces result in my Mullvad WG connection dropping and not automatically re-establishing.  I have to manually disable and enable WG to force a reconnection.  Is this a known issue, or have I configured something incorrectly?

If you could advise me, I would really appreciate it.  I'm probably going to wonder why I was such an idiot once you point out what I should have done.

LAN: 10.17.42.0
OPNSense: 10.17.42.1
WG personal inbound tunnel network: 10.17.50.1
WG personal inbound client: 10.17.50.2

Gateways


Interface Assignments


Firewall Rules: Categories (Any not displayed in detail are due to no rules set in the category)


Firewall Rules: Floating


Firewall Rules: WAN (IDNET)


Firewall Rules: LAN


NAT: Outbound


WG - Mullvad Local


WG - Mullvad Endpoint


WG - Personal Inbound Local


WG - Personal Inbound Endpoint

Can you rephrase in one sentence what you want to achieve and then give some facts (IP's) etc. and screenshot of rules, outbound nat, wireguard config, assigned interface and gateway

Sorry, looking back at my posts, it is a lot of waffle. The result of coming off a long shift, dealing with kids then deciding it's a great idea to bash your tired head against a system into the early hours. I'll tidy up when I get home.

Decided to try and sort the gateway tonight as well, and I have no idea what is going on with this.

Used your guide for reference, as well as the one on the OPNsense How To.  Setup the gateway using the Wireguard interface.  Amended my firewall rule to pass LAN traffic through the gateway only.  Set a NAT for the Wireguard interface to the Mullvad IP assigned to me.

It all worked as expected.  I thought I'd see what happened if I took the WG gateway offline, so I marked it as down.  Traffic continued to pass through.  I disabled the WG gateway, traffic continued to pass through.  Check with Mullvad shows I am still secure through WG, as do third party sites, which confused me a little.  Disabling WG itself finally causes traffic to shunt back through the WAN gateway, I assume as the auto rules kicked in to pass back to the default gateway.  Setting this to skip the auto rules finally cuts off all traffic via the WAN as the firewall rules I have set still define to go via the WG gateway.

Re-enabling everything, traffic is still blocked from the WG gateway if the skip rules box is ticked.  It seems that, with this enabled, the WG gateway is always considered down by the system even if I have ticked for that gateway to always be considered up.  I don't think I've made any errors here, but please advise if I've missed something, or if this is known to currently not work.  To confirm, I did try restarts and flushing states as well, but no effect on this.

I think I've worked it out.  Have been scouring for info where others may be affected and found some comments by other Wireguard users regarding similar behaviour i.e. some sites working fine, others being intermittent, others inaccessible.  It is the MTU and MSS settings, seems the packets flowing through WG are not happy at all about the default sizes and something is preventing the communication to resolve this.  I have forced them manually to 1200 MTU and 1000 MSS, and suddenly everything went back to working, with all traffic flowing to Mullvad and out from there.  I'll have to tinker to find where the cutoff point is.

I'm not sure if this is caused by something I've missed in the config somewhere.  I've gone back to default setup and followed multiple guides, all of which have resulted in the same, so I assume either this just affects all WG, or all WG via Mullvad, or I'm setting/failing to set something which affects the MTU/MSS auto-discovery.  If that's the case, please advise.

The only other issue I need to deal with now is that any changes to the firewall or interface settings drops the WG connection and it doesn't reconnect automatically, so all traffic starts going back through the WAN.  Have to disable and re-enable WG to get it working again.  I'm hoping that this fix may mean the system considers a gateway through my WG interface to be up now, so I can stick some stricter controls in.

EDIT: Further testing, and a setting of 1400 MTU/1360 MSS seems optimal.  Anything higher causes my connection to start falling apart, with around a 50% reduction in download speeds, and intermittent behaviour by sites.  Above 1440/1400 it becomes worse and 1460/1420 or higher is back to mostly unusable.

EDIT AGAIN: Steam remained unstable at 1400/1360.  Dropping to 1380/1340 has resolved that as well.

You tried this?

https://docs.opnsense.org/manual/how-tos/wireguard-client-mullvad.html

I have.  It kind of works but not fully.  I get a WG tunnel established, can see the handshake, and using Mullvad's tools as well as some third parties, I see a secure connection established to Mullvad with an IP I expect via them, no DNS leaks, no WebRTC leaks and IP not blacklisted.  Everything appears secure and as desired.  Firewall logs show lots of traffic passing outbound through wg2 rather than WAN, with a small amount of inbound traffic attempts via WAN and wg2 being denied.  Mostly green and seems nothing out of the ordinary.

The issue I am having with this setup though is that while some websites and connections work perfectly, others are utterly inoperable or intermittent.  Gmail is barely operable, locking up and if loaded various services such as Hangouts etc don't work as expected.  They seem to allow outbound traffic as others receive my messages I post but I don't see anything from them.  BBC News is completely inaccessible, as are a bunch of other sites, while others including this this forum respond perfectly (EDIT: Not perfectly, can login and read but unable to post till I disabled WG and returned to my WAN).  Steam collapses as well, lets me startup and login but the store keeps failing and the friends service cannot connect.

I can ping sites that I cannot connect to, from my various network client devices on wired and wireless.  I can also ping and resolve addresses within OPNsense but that I assume is down to the NAT rule for keeping localhost traffic out via WAN.  However, I have found that even though I can ping the failing sites from my client devices, a traceroute fails to them as it hits Mullvad's WG exit node.  I have tried GB1, 2 and 3, as well as one of their Swedish nodes.  So, is this all working perfectly my end and actually some kind of Mullvad issue with their WG exits/DNS that only affects certain IPs?

mimugmail, I hope you can help please.  I have a fresh OPNsense install using 18.7.9 and am trying to setup Wireguard to connect to Mullvad and push all traffic through the tunnel.  I have followed your Azire guide, substituting where needed for Mullvad's config, but I just cannot get it to work as expected.  It seems that the system treats the Mullvad gateway as perpetually down, instead pushing all traffic to my WAN, and I cannot work out why.  If I tick to skip rules when the gateway is down, I see only a wall of blocking in the firewall logs.  If I leave it unticked, all traffic goes out via my raw WAN connection instead.  I have tried various other guides too and none of them work.

Is it that something has changed or broken in 0.8_1 please, or am I doing something wrong?  I have successfully setup WG connections from my Android devices back to my LAN with no issue, so I assumed this wouldn't be much harder, but it has me stumped so far.  My final goal is to be able to have my external devices able to connect to my LAN and all traffic from my LAN and any connecting devices to go back out via Mullvad.