Some troubles with my wireguard setup - changes between 19.1 and 19.7?

[Dark-Sider]

Hi,

to set the general picture: I installed wireguard devel on 19.1.x a couple of weeks ago. As I now had time play around with my planned wg VPN tunnel I finally did so yesterday.

I already have multiple ipsec and openvpn tunnels on my opnsense box running - so I thought this would be an easy and straight forward task :-)

The purpose for my setup is to route some hosts of my network through an VPS server in canada. This VPS runs ubuntu 18.04 with wireguard. I did manage to bring up the tunnel quite easily. The tunnel network is 192.168.4.0/24 with my wireguard box being .2 and the VPS .1

Without defining any other virtual interfaces (just setting the allow rule for the wirguard interface) I was able to ping the remote location - ping times and tcpdump also show that it actually is the VPS who answered.

My first try was to put 192.168.4.0/24 as "allowed ips" in the wireguard config on the opnsense box. Once the tunnel was established the routes were set and the tunnel worked.

The next step was to route actual internet upstream traffic through the vpn. I looked in the forum and found those threads:
https://forum.opnsense.org/index.php?topic=8998.0 (ovpn)
https://forum.opnsense.org/index.php?topic=4979.0 (ovpn)
https://forum.opnsense.org/index.php?topic=11737.0 (wg)

So I went ahead and did
- define an alias for one host in my network,
- setup the NAT rules (but not needed as NAT can be achieved on the VPS, just makes allowed ips config easier)
- created the virtual VPN interface with assigned wg0 to it and set it to ipv4 DHCP
- created the firewall rule with the alias and the gateway of the virtual VPN interface.

It did not work as I had expected. tcpdump on the VPS shows, that 1962.168.4.0/24 traffic is sent through the tunnel to the VPS but all other traffic just doesn't go through the tunnel - but also does not exit my opnsense on any other interface (ping shows a timeout).

I did a bit of thinking and replaced the allowed ips in the wg config with 0.0.0.0/0 as this might be the problem.

After restarting  the box my whole network was routed through the VPN not just this one host. However, the traffic actually went over the VPS in Canada. It took me some time to figure out that I probably had to check that disable routes checkbox in the wireguard config, das my default route was now set to 0.0.0.0/0 -> VPN

before doing so however I also read that 19.7 was now released and has wg as stable on board. So I reset my opnsense VM to a config state before my wireguard efforts and upgraded the box to 19.7.2 (uninstalled wg-devel and installed the stable wg package)

After setting up the wg tunnel again and verifying that the VPS is pingable from my LAN I again defined the virtual interface.

Then things changed.

First thing I noticed is, that there is no more gateway created automatically once you choose IPV4 DHCP for the virtual interface --- why?

I just stayed with ipv4 "none". Restarted  the tunnel and the VPN interface got the correct ip 192.168.4.2 - a minor change I thought.

I created the gateway manually leaving all IP fields blank just setting the Interface to VPN. After pressing OK and Applying there was no gateway added. I retried this a couple of times - no error message - no gateway.

I then created the gateway with the IP 192.168.4.1 and Interface VPN - which actually worked.

I thend changed the allowed ip again to 0.0.0.0/0 - and all traffic was again forced through the tunnel.

After that I ticked the "disable routes" option in the wg config and restarted the box (just to be safe). This is when things got really bad.

The default route was gone, however so was any route pointing 192.168.4.0/24 to go out through the VPN interface. Thus, 192.168.4.1 (my VPS) was no longer reachable. The routing table also shows only 192.168.4.2 (my opnsense box) on the VPN interface without any netmask.

I then tried to add the route manually as read on the forums, however this is not (not anymore?) possible. Whenn adding the route I can't chose my VPN interface. I only have choice of my PPPoE gws for my dial-up connection.

This is where I hit the roadblock, as I was not able to set
- 0.0.0.0/0 as allowed ips
- without setting a default route through the tunnel
- and I failed adding the 192.168.4.0/24 route manually

Were there changes in 19.7 how this is done?
Is there an obvious mistake I just don't see?

Thanks for helping out!

bye
Fabian

[mimugmail]

Some mistakes, yes.
First, don't use DHCP in Wireguard, neither 19.1 nor 19.7. Gateways are automatically added when type is DHCP, but Wireguard doesn't do DHCP, so you MUST add one manually. This is all perfectly documented here:
https://docs.opnsense.org/manual/vpnet.html#configuration

With 19.7 gateway code changed (not Wireguard Code). You have to verify that your default gateway has "upstream" checked and your Wireguard gateaway not. Then this thing can't happen.

[Dark-Sider]

Hi,

thanks for your swift reply. I changed the interface from DHCP to ipv4 none. I also tried to add a gateway like in the guide:

To do this, go to System ‣ Gateways ‣ Single and add a new gateway. Choose your WireGuard interface and set the Gateway to dynamic.

If I do so and put "dynamic" in the gateway's ip address field the gateway just won't appear in the list. Also there is no errormessage. If I put in the ip adress of my VPS server's tunnel interface (192.168.4.1) the gateway shows up. Maybe this is part of the new gateway code.

I already put my ISP's v6 and v4 gateway as upstream gateway, the vpn_gw does not have that option set.

As for the routing stuff, I'll check this once I'm home tonight. When I  check "disable routes" I now successfully created the route to the tunnel via commandline:

route add -inet 192.168.4.0/24 -link -iface wg0
Is there any way to make this persistant via the gui?

while I did so I noticed another strange thing:

netstat -r lists the following:
192.168.4.2        link#13            UH          wg0
While the .2 IP-Adress is the local ip of my wg interface it is routed out via wg0. If I ping 192.168.4.2 from my opnsense box, packets go into the tunnel, and come back after 180ms (from canada):

 root@OPNsense:~ # ping 192.168.4.2
PING 192.168.4.2 (192.168.4.2): 56 data bytes
64 bytes from 192.168.4.2: icmp_seq=0 ttl=63 time=188.488 ms

traceroute to 192.168.4.2 (192.168.4.2), 3 hops max, 40 byte packets
 1  192.168.4.1 (192.168.4.1)  94.379 ms  94.387 ms  94.121 ms

If I ping 192.168.4.2 from a device from my LAN it shows my local reply times <1ms

the local ip address of an openvpn tunnel however is bound to lo0
10.41.0.2          link#11            UHS         lo0

is this intended?`

With 19.7 gateway code changed (not Wireguard Code). You have to verify that your default gateway has "upstream" checked and your Wireguard gateaway not. Then this thing can't happen.

So you would say, that ticking the "disable routes" box is not necessary anymore with 19.7, when the ISP gateways have set the upstream flags and the vpn-gw does not?

regards,
Fabian

[Dark-Sider]

some additional input:

if I uncheck disable routes mit routing table looks like this:

Code: [Select]

Internet:
Destination        Gateway            Flags     Netif Expire
0.0.0.0/1          wg0                US          wg0
default            ppp-default.m-onli UGS      pppoe0
[...]

and all traffic is sent through wg0 - upstream flags are set correctly

[sashxp]

Hi,

thanks for your swift reply. I changed the interface from DHCP to ipv4 none. I also tried to add a gateway like in the guide:

To do this, go to System ‣ Gateways ‣ Single and add a new gateway. Choose your WireGuard interface and set the Gateway to dynamic.

If I do so and put "dynamic" in the gateway's ip address field the gateway just won't appear in the list. Also there is no errormessage. If I put in the ip adress of my VPS server's tunnel interface (192.168.4.1) the gateway shows up. Maybe this is part of the new gateway code.

Like me you are going into the same Bug - pls look at https://forum.opnsense.org/index.php?topic=13612.msg63257#msg63257 - there is already a patch.

I think we are having the same error: https://forum.opnsense.org/index.php?topic=13713

[mimugmail]

@Dark-Sider:
So, you set the gateway IP to the real IP instead of "dynamic" and tick "disable routes" and have what exactly???

[Dark-Sider]

@Dark-Sider:
So, you set the gateway IP to the real IP instead of "dynamic" and tick "disable routes" and have what exactly???

I set the gateway IP to the real IP so I'm not affected by that bug that won't let me add the gateway. I personally would prefer the dynamic solution. Since I control both endpoints of this setup the gateway's IP adress won't suddenly change so this setup works for me right now.

I have to tick disable routes since I have to add 0.0.0.0/0 in the wireguard's allowed list. If I don't tick it - a new route with highest priority is added to opnsense once the tunnel is established and all traffic goes down that new route. If this should not happen anymore when the other gateways have that "upstream" box ticked, than there is still a bug present, as it happens to me in 19.7.2

Without ticking the box and adding the route for the tunnel network manually on console my setup starts working as it is supposed to be.

So a option, what kind of routes should be added once the tunnel IF goes live would be cool. Usually this would be just the route for the tunnel-network.

regards,

Fabian

[mimugmail]

I have a talk to the devs .. hopefully we find a solution.

[mimugmail]

Ok, I think there's a way to go, but no implementation to the gui yet:

1. setup like usual
2. tick "Disable routes"
3. go to CLI and do a "route add <gwip> -iface wgX" (most important it lies in the range to "Allowed IPs")
4. then add a gw for the assigned interface (ipv4 none) with the IP you added before
5. set the firewall rule you want and add the gateway

I'll try to add a new field to the UI which let's you add this IP so action 3. is not needed, but I need some time for it

[spectrely]

So I'm having a tough time following this.

I followed the OPNsense documentation guide, and I can create the tunnel, but I can't access my internal network or the internet.

Local Configuration:
Name: HomeCloud
Public Key: <Server Public Key>
Private Key: (hidden)
Listen Port: 51820
DNS Server: 192.168.20.1
Tunnel Address: 192.168.70.1/24
Peers: <Client 1>
Disable Routes: <Checked>

Endpoint:
Name: <Client 1>
Public Key: <Client 1 Public Key>
Allowed IPs:
192.168.70.2/24 - <Client 1 Address>
192.168.10.0/24 - <VLAN For Other Stuff>
192.168.20.0/24 - <VLAN For Other Stuff>
192.168.30.0/24 - <VLAN For Other Stuff>
192.168.40.0/24 - <VLAN For Other Stuff>
192.168.50.0/24 - <VLAN For Other Stuff>
192.168.60.0/24 - <VLAN For Other Stuff>
192.168.66.0/24 - <VLAN For Other Stuff>
(When I put 0.0.0.0/0 all my connectivity stops)

List Configuration Output:
interface: wg0
  public key: <Server Public Key>
  private key: (hidden)
  listening port: 51820

peer: <Client 1 Public Key>
  allowed ips: 192.168.10.0/24, 192.168.20.0/24, 192.168.30.0/24, 192.168.40.0/24, 192.168.50.0/24, 192.168.60.0/24, 192.168.66.0/24, 192.168.70.0/24
  persistent keepalive: every 25 seconds

Client Settings (Phone):
Interface
Name: HomeCloud
Public Key: <Client Public Key>
Addresses: 192.168.70.2/32
DNS Servers: 192.168.20.1

Peer
Public Key: <Server Public Key>
Endpoint: vpn.example.com:51820
Allowed IPs: 192.168.70.0/24
Persistent Keepalive: 25

Firewall
NAT -> Port Forward
WAN   UDP   * *   WAN address   51820   192.168.70.1   51820   Wireguard_Inbound

NAT -> Outbound
WAN   WireGuard net   *  *  *   Interface address   *   NO   Wireguard_Outbound

Rules -> WAN
IPv4   UDP  *  *   192.168.70.1   51820  *  *   Wireguard_Inbound

Interfaces
No interface setup for wg0 (I got confused here)

System -> Gateway -> Single
No gateway set, I don't understand the way this and the interfaces interact.

1. setup like usual
2. tick "Disable routes"
3. go to CLI and do a "route add <gwip> -iface wgX" (most important it lies in the range to "Allowed IPs")
4. then add a gw for the assigned interface (ipv4 none) with the IP you added before
5. set the firewall rule you want and add the gateway

1) I set it up like the above.
2) Clicked disable routes,
3) Went the the CLI and ran

Code: [Select]

route add 192.168.70.1/24 -face wg0Not sure which list of allowed IPs I'm supposed to adhere to.
4)
- Tried to add the gateway, needed to link to an interface.
- Added an interface with IPv4 None, went and tried to add a gateway, complained of no IP on the interface.
- Put an IP of 192.168.70.1 on the interface, then tried to add gateway. Same message.
- I deleted the interface on wg0 and the gateway I created.
- Tested connectivity with what I had without the gateway and interface, handshake successful, no internet/network access. For testing I have allow in/out rules on all interfaces except WAN.

At this point I'm completely lost. Can anyone help?


Respectfully,

Spectrely

[mimugmail]

Why do you want to set 0.0.0.0 on OPNsense? This would make it as a Client. All these networks defined, are they behind OPN or client?

[spectrely]

I wanted 0.0.0.0/0 on both client and server because I thought that the client wouldnt be able to access other network resources without it
Thanks for correcting that.

All of the additional subnets are configured on server side. I have my network segregated by subnet/vlan by purpose. e.g.

192.168.20.0/24 = Cloud Applications
192.168.66.0/24 = VLAN 666 or DMZ
192.168.20.0/24 = Hypervisor traffic
So on and so fourth.
I would like my clients to really just connect to the *.*.20.0 subnet, and use that gateway to also get internet. SO I can run things like bitwarden and nextcloud without exposing them to the internet.

However, for now I just want to get this working and understand it better.

Thanks!

[mimugmail]

On Server Side, in endpoint config, just put Tunnel IP with /32. On Client Side in Tunnel IP all the networks or 0.0.0.0/0

[spectrely]

I did that, and I still can't access external resources or the internet.

[spectrely]

Updated Config:

Local Configuration:
Name: HomeCloud
Public Key: <Server Public Key>
Private Key: (hidden)
Listen Port: 51820
DNS Server: 192.168.20.1
Tunnel Address: 192.168.70.1/32
Peers: <Client 1>
Disable Routes: <Checked>

Endpoint:
Name: <Client 1>
Public Key: <Client 1 Public Key>
Allowed IPs:
192.168.70.1/32 - <Tunnel Address>
192.168.70.2/24 - <Client 1 Address>

List Configuration Output:

Code: [Select]

interface: wg0
  public key: pSU90xYiwwv/K23GOnqKVUoGtw9YH+WuvC3tnxuyoEk=
  private key: (hidden)
  listening port: 51820

peer: hDB112sXsxc9JpO4QbvoshrRiEZDwZ4idUv9ls3nyzk=
  endpoint: 174.226.7.136:3348
  allowed ips: 192.168.70.1/32, 192.168.70.2/32
  latest handshake: 2 minutes, 15 seconds ago
  transfer: 1.35 KiB received, 284 B sent
  persistent keepalive: every 25 seconds
Client Settings (Phone):
Interface
Name: HomeCloud
Public Key: <Client Public Key>
Addresses: 192.168.70.2/32
DNS Servers: 192.168.20.1

Peer
Public Key: <Server Public Key>
Endpoint: vpn.example.com:51820
Allowed IPs: 0.0.0.0/0
Persistent Keepalive: 25

Firewall
NAT -> Port Forward
WAN   UDP   * *   WAN address   51820   192.168.70.1   51820   Wireguard_Inbound

NAT -> Outbound
WAN   WireGuard net   *  *  *   Interface address   *   NO   Wireguard_Outbound

Rules -> WAN
IPv4   UDP  *  *   192.168.70.1   51820  *  *   Wireguard_Inbound

Interfaces
No interface setup for wg0 (I got confused here)

System -> Gateway -> Single
No gateway set, I don't understand the way this and the interfaces interact.