OPNsense Forum
English Forums => General Discussion => Topic started by: SomethingOrOther on July 10, 2020, 10:00:21 pm
-
Hello,
I have a strange port forwarding issue.
I'm running two OPNSense routers cascaded together. I know, it's not recommended, but that's the setup I need to work with.
So the WAN IP from the 2nd OPNSense is on the first one's LAN. And everything works.
Initially I was doing double NAT on router 2, but I created a static route on router 1 to reach the networks on router 2. It worked, all good. Disabled Outbound NAT on router 2. Added NAT rules for that traffic on router 1. All good. Everyone can access everything they need to access, according to their firewall rules. Great.
But, on router 2, I have a couple of OpenVPN servers, an IPSec server and a WireGuard "server" running as well. To access these from the outside, I need to set up port forwarding rules. I set up the rules on router 1 like this:
Source: Any / Destination: WAN Address / Source Ports: Any / Destination Port: whatever port the server runs on / NAT IP: IP of the server / NAT Ports / whatever port the server runs on.
And this automatically adds the corresponding Firewall rule on WAN.
So, for OpenVPN & IPSec, it just works. I can connect without issue from outside and access everything the firewall rules allow me to access. But for WireGuard, the traffic doesn't return to the appropriate host and I can't access anything from my client device (no Internet, no local networks). The handshake never completes. I can see the WireGuard instance on router 2 receives the incoming packets but I assume it can't send them back using the appropriate route.
If I do a "double port forward", meaning from router 1 I forward the outside traffic to the LAN IP which is router 2's WAN IP. And then on router 2 forward that traffic to the actual host on router 2, everything works. But if I don't do the double port forward and set the single port forward up as I did with OpenVPN & IPSec, it breaks.
I can see form the Firewall logs that neither router 1 or router 2 is not blocking the WireGuard traffic. I can see the traffic being passed in the logs. But I think it doesn't understand where to send the packets back and that's why it fails.
But I'm pretty much at a loss as to how to figure out where the traffic is going and what I need to do for it to route properly.
Any help would be appreciated - even just hints would be great.
Apologies in advance if I forgot to add important information. Just ask me and I will provide whatever is needed.
Cheers
-
It seems the pf code and WireGuard interfaces dont like each other. Hast similar issue with another guy and Mullvad
-
Not the resolution I was hoping for, but at least I know it isn't me... Thanks.
-
i think, i was the guy with WG and Mullvad - here is my thread: https://forum.opnsense.org/index.php?topic=17973.0
Did you try 20.7 Beta? Perhaps the issue is fixed there?
-
Hey there,
Thanks for chiming in.
I'm not in a position where I can install a beta on these systems. So I haven't tried that, no. But, in my opinion, it would have more to do with the WireGuard package than OPNsense itself.
I'm no expert, but there has to be something different in how WireGuard routes traffic. The port forward & firewall rules I've set up are correct (as far as OpenVPN, IPSec and anything else) - just not WireGuard. So I'm at a complete loss.
As far as your case is concerned, are you cascading routers or is it just a "regular" port forward that you can't get to work?
-
It's some pf magic which is missing or not working. Quite sure it will also not work with 20.7
-
When you say "pf magic", you mean it's an issue with FreeBSD?
-
I have no idea .. I'd guess some missing or wrong rules
-
Hey mimugmail,
Sorry but I'm a bit confused. I interpreted your first response as stating that it's an issue with the code somewhere and was not related to my configuration. But your last response makes me think that it is...
Could you clarify what you mean? :-) If it's me, I'll keep trying.
-
Again, I have no idea ;) It could be a missing gateway rule or something which is different to other things
-
Hi SomethingOrOther,
did you already Chester an Github Issue?
Perhaps someone from the Developers could look into this?
Sash
-
Hey sashxp,
I guess I could file a bug report, but I'm not sure anyone would invest much time on such a fringe issue.
I also believe that my problem is related to having two routers cascaded and that I'm port forwarding through the two routers over a static route, in order to avoid double NAT. I think the problem is that the WireGuard instance doesn't know where to send back the packets it receives from the client and the handshake never completes. Why it works perfectly with OpenVPN & IPSec but not WireGuard is the mystery...
A single port forward from the router on which the WireGuard instance is running (not the edge network) works just fine. It's only forwards from the outside over the two routers that fail with WireGuard (though, again, it works perfectly with OpenVPN and IPSec).
ALso, I have a perfectly useable workaround: two port forwards, one on each router.
It's more about me being curious and rather anal than anything else... ;-)
Your issue seems to relate to your commercial VPN provider forwarding a port for you on their VPN network so that you can access your internal networks while connected to their VPN. So I'm not convinced we're experiencing the same issue.
-
I have a similar issue, thread: https://forum.opnsense.org/index.php?topic=18062
Wireguard and stateful connections outside of the wg interface address range dont seem to work ? Opnsense routes the ack reply over the default gateway instead of wg0 where it came from. I tried overriding the gateway with several pbr but is has no effect. Or am i missing something ?
Who is responsible for stateful connection tracking, wireguard plugin or opnsense ?
-
i've tried this evening if there is a change in v20.7 but had no luck. Everything regarding this issue is the same :-(
I think we need a bug report on github to get a fix?
@SomethingOrOther or @Voodoo could one of you report an issue on github? I don't think we get a fix if we don't create an issue... :-(
-
As everyone here seems to have a little bit different issues, i would really like to know if the problem in your cases is also that opnsense routes the reply over the wrong gateway.
@sashxp could you do a packet trace on the opnsense (check lan, wg0, and wan interface) and try the port forwarding, do requests get send over the wan gateway back as well ?
I cant believe that nat with wireguard is not working, if that was the case there should be much more people with the problem, or nobody is trying out wireguard. Or maybe they use it only as road warrior vpn, which works fine.
Im a developer and not that deep into networking, its just for homelab use to host some websites i work on. So there could be a misunderstanding on my side about routing and nat.
-
Today I fixed an issue where reply packet was sent via OpenVPN ofvsome other vpn Provider. Best would be to stop or disable all other vpn stuff to find the issue
-
in my case i did not have openvpn or ipsec enabled during my wireguard tests.
Only a wireguard interface and some vlans.
-
FYI, I ran into the same issue and opened a bug report on github, if anyone wants to chime in
https://github.com/opnsense/core/issues/4389
-
I have similar Problems.
I have 2 PPPoe Wan Adapter (1 for Voice and 1 for Filehosting) an 1 Default Gateway for nomal Internet surfing.
When i only open the Port to WG on the PPPoe for Fileosting i see the Traffic came in an that WG send some traffic out. But he didnt sent it back over the interfaces there it came from. He sends it over the Default GW (see it in tcpdump).
So the Handschake dont work.
-
Can you try if this also happens with -kmod?
-
Hi,
what does -kmod do?
-
It uses ifconfig to create interfaces and the -go variant uses its own one. With -kmod e.g. also CARP works compared to -go
-
Sorry for my question.
Does kmod mean to install it by hand and not through the sense GUI?
-
Yes, pkg install wireguard-kmod
-
Ok. But i think if i install this, i cant use the gui anymore. Is that right?
-
Just Install an reboot, you still use the old Plugin and as backend the kmod