WireGaurd Public IP traffic replying out of wrong WAN.

[FingerlessGloves]

Hi Guys,

I'm not sure if this is a bug with reply-to or misconfiguration somewhere. I've had others doing this same setup and get the same issue, the traffic returns out of the WAN instead of back over the WireGuard tunnel.

The Setup

OPNsense 1 is in a DC with two public IPs say 51.51.51.1 and 51.51.51.10.
The WAN interface is 51.51.51.1, with a gateway of 51.51.51.254.
51.51.51.10 has been set as a Proxy ARP virtual IP on the WAN interface.
I have then created a WireGuard local on this OPNsense with the IP of 10.0.0.1, I then added a WireGuard peer of another OPNsense box (OPNsense 2). The AllowedIPs of this peer is just 51.51.51.10.
This WireGuard tunnel interface is named "WG_RoutedIP"
I have then created a WAN rule to allow any traffic to 51.51.51.10 to pass, and I also created a ANY to ANY rule on the WG_RoutedIP interface.

OPNsense 2 at home or office etc
WAN interface is 31.31.31.1 with gateway of 31.31.31.254
I have then created a WireGuard local on this OPNsense with the IP of 51.51.51.10, I have Disable Routes ticked and then a gateway of 51.51.51.254 set.
I then added a WireGuard peer of another OPNsense box (OPNsense 1). The AllowedIPs of this peer is 0.0.0.0/0.
This WireGuard tunnel interface is named "WAN_RoutedIP"
I have then create gateway on WAN_RoutedIP using 51.51.51.254 as the far gateway and corresponding NAT rule.

If you set a client to use this new routed IP WAN, as a gateway using a gateway policy rule, the traffic works and I can browse the internet fine as 51.51.51.10.

If you allow the HTTPS WebUI or do a port forward then try browse to 51.51.51.10 from a PC that's not be hide either OPNsense box, you can see the traffic go through OPNsense 1, then hit OPNsense 2. Client can not connect, so I then looked at the traffic going over the WAN of OPNsense 2 and I can see the return traffic is exiting out of the WAN not WAN_RoutedIP, so this would point to reply-to not being enabled but it is.

If I run pfctl -s all, I believe the reply-to rule is there if I'm looking at the right thing
```
pass out log route-to (vtnet0 31.31.31.254) inet from 31.31.31.1 to ! (vtnet0:network) flags S/SA keep state allow-opts label "2f613a9ac318a59b487c1251230f5a27"
pass out log route-to (wg1 51.51.51.254) inet from 51.51.51.10 to ! (wg0:network) flags S/SA keep state allow-opts label "6dd6ab373ac72f668fb2f29d408b0231"
```

Note: IPs have been changed to simplify the setup and show clear distinctions.

Hopefully someone knows what's going on here!

FingerlessGloves

[optic]

i have been on this problem for a while now and its one big reason i cannot sunset two of our commercial firewalls at work (yet) and switch them over to opnsense because i need to route public v4s from a datacenter to a server.

[Gauss23]

i have been on this problem for a while now and its one big reason i cannot sunset two of our commercial firewalls at work (yet) and switch them over to opnsense because i need to route public v4s from a datacenter to a server.

And what has this to do with WireGuard? WireGuard is not yet production ready in my opinion. At least not for something like a business site-to-site tunnel. No commercial firewall has WireGuard implemented and there is no sign that one of the big players is planning in doing so.

In my opinion OpenVPN site-to-site are the most stable solution for OPNsense currently. IPsec is ok, too, but I heard about strange problems doing NAT on the tunnels. In some cases that's needed.

[FingerlessGloves]

I'm 50/50 if WireGuard should be used in production when its in its GO version, but once its in kernel space, should get less issues, as its sorta native to the OS then.

Anyway please try keep this topic on topic. I don't believe its just WireGuard that's effected by this odd issue with the reply-to. I think I read in IRC someone else tried it with GRE and got same results but this was months ago now.

If other can replicate the issue, confirm my findings and if they are the same, then something is up.

```
pass out log route-to (vtnet0 31.31.31.254) inet from 31.31.31.1 to ! (vtnet0:network) flags S/SA keep state allow-opts label "2f613a9ac318a59b487c1251230f5a27"
pass out log route-to (wg1 51.51.51.254) inet from 51.51.51.10 to ! (wg0:network) flags S/SA keep state allow-opts label "6dd6ab373ac72f668fb2f29d408b0231"
```

I was looking at the pf rules and I wondered how do you expand (wg0:network) or way of finding what its expanding it too? cause if its applying that route to traffic not going too (wg0:network), then does expand is going to something like 0.0.0.0/0, which would be the rule wouldn't apply. In theory it should expanding too 51.51.51.10/32 🤔

[Maurice]

If I run pfctl -s all, I believe the reply-to rule is there if I'm looking at the right thing

Maybe not.

Code: [Select]

pass out log route-to (wg1 51.51.51.254) inet from 51.51.51.10 to ! (wg0:network) flags S/SA keep state allow-opts label "6dd6ab373ac72f668fb2f29d408b0231"

'route-to' is different from 'reply-to'. This rule allows outgoing connections and routes them via the specified gateway.
There should also be a 'pass in' rule which allows incoming connections on the "WAN_RoutedIP" interface. And this rule should have the 'reply-to' tag.

[FingerlessGloves]

'route-to' is different from 'reply-to'. This rule allows outgoing connections and routes them via the specified gateway.
There should also be a 'pass in' rule which allows incoming connections on the "WAN_RoutedIP" interface. And this rule should have the 'reply-to' tag.

Ah I wasn't quite sure but that makes more sense about route-to and reply-to, thank you for the clarification

I've now gone over all the rules and looked for reply-to which isn't there.

[optic]

Issue is the same with GRE Tunnel. It seems to be with all virtual Interfaces.

[Maurice]

As far as I know, reply-to should work if

• it is globally enabled (advanced firewall settings) and
• the interface has an upstream gateway assigned to it (interface settings) and
• there are 'allow in' firewall rules on the interface. These rules should then have the reply-to tag added automatically.

If it still doesn't work, it might indeed be an issue with virtual interfaces.

[FingerlessGloves]

Although when you create OpenVPN or WireGuard interfaces and assign them you leave the IPv4 config as None and it then gets assigned by the service.

So if the upstream gateway selecting on the interface is required, how do you get around this?

[FingerlessGloves]

I've gotten it working 

In the interface setting for the WG0. Instead of having it set as none like you do normally for WireGuard.
Setting the IPv4 Address and the correct IPv4 upstream gateway, boom it started working.

Rules now the reply-to
```
pass in log quick on wg1 reply-to (wg1 51.51.51.254) inet proto tcp from any to 192.168.1.20 port = rdp flags S/SA keep state label "b13b654e93dd92492f867ba7d182bce8"
pass in log quick on wg1 reply-to (wg1 51.51.51.254) inet proto icmp all keep state label "0e99bf94860f81e34a3b417ce79f8d77"
```

Is this how I should be doing it with WireGaurd, because of it also wanting to set the interface address when the wireguard-go service starts. Could this cause issues long term?

I did try ticking "Dynamic gateway policy" and setting "IPv4 Configuration Type" to none, but this doesn't look to generate the reply-to rules required.

[mimugmail]

It should work stable when it doesnt change, nice idea

[Maurice]

Great to hear that it works! Makes sense. No upstream gateway, no reply-to. A different solution (without configuring the wg interface statically) would probably require code changes.

[FingerlessGloves]

Great to hear that it works! Makes sense. No upstream gateway, no reply-to. A different solution (without configuring the wg interface statically) would probably require code changes.

Me too  , good learning curve if anything.

Yeah I wonder if we need a dynamic upstream gateway option to go with the "Dynamic gateway policy", but I guess that's up to the dev's.