Show Posts
16
Development and Code Review / Re: Wireguard in opnsense
« on: January 15, 2019, 09:15:52 pm »
I finally managed it, but it is a Frankenstein's monster of a solution so far.
From my frantic reading and attempting to learn some more networking, I was under the impression that with static routes a packet would be sent to the one that most appropriately matches i.e. a packet for 8.8.8.8 would go through a route set to that IP, even if a route for 0.0.0.0/0 was before it. I've seen people confirming this with their own setups. But this has been confusing me, as my NAT and firewall rules appear correct and they work for both my outbound VPN provider and inbound remote client VPN just fine, I just can't get both tunnels up at the same time. If I change my outbound VPN to something other than 0.0.0.0/0 then my inbound VPN immediately establishes, but I lose my outbound VPN. I tried running without WG at all, then with each independently, then with both together, and examining the defined routes each time, especially the static routes WG established (they get set automatically in OPNsense). They seemed correct and, as each worked independently, I assumed should still work together. So I disabled OPNsense's auto-static route option and tried defining them manually myself. This still didn't work. I'm not sure if there's a difference in how WG interacts with static routes which is causing this.
So I went to an extreme solution. I copied every assigned /8 IPv4 address block from Wikipedia at https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_address_blocks into a spreadsheet. I first removed private address blocks 10.0.0.0/8, 172.0.0.0/8, 192.0.0.0/8, and my VPN provider endpoint which falls under 185.0.0.0/8, and added 127.0.0.0/8. I sorted them in order then wrote a formula to concatenate them all and separate with a comma. I copied this into the OPNsense config field so that they were recognised as individual address blocks. Saved and started up both WG tunnels. And both immediately established, with my LAN having internet access through the VPN provider as well as my mobile able to access my LAN and then from there the internet through the VPN provider.
It's a hell of a bodge. My states table looks like a crumbling ruin with 240 separate entries to cover nearly every /8 address block and it's not the quickest of systems, as I assume it's now having to parse all of that before each packet is routed, but it works. I don't know why the earlier attempts didn't. Perhaps it is an issue with OPNsense or its WG implementation in how the static routes and associated packets operate, or there's still some config nuance I haven't grasped, or something else entirely. I'm going to continue testing and try to work out what exactly is going on, as I don't feel this is a viable long-term solution and I'm hoping that at least knowing it can work will help something click in my head to get a proper config together. I've seen mention of fwmark and I think that may be what I am missing so far: a means to exclude certain packets from going into a WG tunnel, as that seems a cleaner way to do this rather than have to strictly define most of the internet to go through the tunnel. There's no exclusion option in the OPNsense GUI, so I'll have to try and see if this can be made to work manually on the CLI.
If I get it working without it being the abomination above, I'll let you know and get some screengrabs along with a clearer description.
From my frantic reading and attempting to learn some more networking, I was under the impression that with static routes a packet would be sent to the one that most appropriately matches i.e. a packet for 8.8.8.8 would go through a route set to that IP, even if a route for 0.0.0.0/0 was before it. I've seen people confirming this with their own setups. But this has been confusing me, as my NAT and firewall rules appear correct and they work for both my outbound VPN provider and inbound remote client VPN just fine, I just can't get both tunnels up at the same time. If I change my outbound VPN to something other than 0.0.0.0/0 then my inbound VPN immediately establishes, but I lose my outbound VPN. I tried running without WG at all, then with each independently, then with both together, and examining the defined routes each time, especially the static routes WG established (they get set automatically in OPNsense). They seemed correct and, as each worked independently, I assumed should still work together. So I disabled OPNsense's auto-static route option and tried defining them manually myself. This still didn't work. I'm not sure if there's a difference in how WG interacts with static routes which is causing this.
So I went to an extreme solution. I copied every assigned /8 IPv4 address block from Wikipedia at https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_address_blocks into a spreadsheet. I first removed private address blocks 10.0.0.0/8, 172.0.0.0/8, 192.0.0.0/8, and my VPN provider endpoint which falls under 185.0.0.0/8, and added 127.0.0.0/8. I sorted them in order then wrote a formula to concatenate them all and separate with a comma. I copied this into the OPNsense config field so that they were recognised as individual address blocks. Saved and started up both WG tunnels. And both immediately established, with my LAN having internet access through the VPN provider as well as my mobile able to access my LAN and then from there the internet through the VPN provider.
It's a hell of a bodge. My states table looks like a crumbling ruin with 240 separate entries to cover nearly every /8 address block and it's not the quickest of systems, as I assume it's now having to parse all of that before each packet is routed, but it works. I don't know why the earlier attempts didn't. Perhaps it is an issue with OPNsense or its WG implementation in how the static routes and associated packets operate, or there's still some config nuance I haven't grasped, or something else entirely. I'm going to continue testing and try to work out what exactly is going on, as I don't feel this is a viable long-term solution and I'm hoping that at least knowing it can work will help something click in my head to get a proper config together. I've seen mention of fwmark and I think that may be what I am missing so far: a means to exclude certain packets from going into a WG tunnel, as that seems a cleaner way to do this rather than have to strictly define most of the internet to go through the tunnel. There's no exclusion option in the OPNsense GUI, so I'll have to try and see if this can be made to work manually on the CLI.
If I get it working without it being the abomination above, I'll let you know and get some screengrabs along with a clearer description.
