Hello, upgrading to OPNsense 23.7.3 has broken routes with wireguard. Connections get established ok, but routing fails with the following errors. My setup has not changed and has been working on prvevious releases. I have configured it according to https://docs.opnsense.org/manual/how-tos/wireguard-client.html (https://docs.opnsense.org/manual/how-tos/wireguard-client.html) From the wireguard log:
2023-09-07T22:23:26 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T22:23:26 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T22:23:26 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T22:23:26 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T21:44:01 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T21:44:01 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T21:44:01 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T21:44:01 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T21:43:16 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T21:43:16 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T21:41:18 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T21:41:18 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T21:41:18 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T21:41:18 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T21:00:01 Notice wireguard Wireguard interface PIA (wg3) started
2023-09-07T21:00:01 Notice wireguard Wireguard interface PIA (wg3) stopped
2023-09-07T20:56:48 Notice wireguard Wireguard interface PIA (wg3) started
2023-09-07T20:56:48 Notice wireguard Wireguard interface PIA (wg3) stopped
2023-09-07T20:56:48 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T20:56:48 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T20:56:48 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T20:56:48 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T20:56:48 Notice wireguard Wireguard interface wgScarnet (wg1) started
2023-09-07T20:56:48 Notice wireguard Wireguard interface wgScarnet (wg1) stopped
When you run this:
# /sbin/route -n add -'inet' '10.7.0.2/24' -interface 'wg2'
It probably says the route already exists?
I'm not sure it's related to your issue, but I can see the problem with "-q" muting the error message (but why?).
Cheers,
Franco
Thanks Franco, it does say it already exists:
root@OPNsense:~ # /sbin/route -n add -'inet' '10.7.0.2/24' -interface 'wg2'
add net 10.7.0.2: gateway wg2 fib 0: route already in table
Its not an issue with the script itself ? "/usr/local/opnsense/scripts/Wireguard/wg-service-control.php" (I have not modified it)
Thanks Franco, when I remove -q from the script command /sbin/route -n add I get the already exists error:
2023-09-08T10:15:34 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was 'add net 10.7.0.3: gateway wg2 fib 0: route already in table'
2023-09-08T10:15:34 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was 'add net 10.7.0.2: gateway wg2 fib 0: route already in table'
Ok, now we don't see this on our end. Do you maybe have a static route set for this previously? It would explain why this fails. And I assume this is after a clean reboot on 23.7.3?
Cheers,
Franco
Thanks Franco, I managed to resolve it by completely removing the interface and associated server and peer configurations, rebooting, then recreating the interface and server/peer conf.
I have the same problem.
When I run without -q, I get:
add net 10.1.2.3: gateway wg1 fib 0: route already in table
I removed the WireGuard twice, even uninstalled the plugin. Didn't help.
However, I don't think running this is necessary as the route is there as it probably got there from the "Tunnel Address" field, which has the CIDER and this is correct:
root@OPNsense:~ # route show 10.1.2.1
route to: OPNsense
destination: OPNsense
fib: 0
interface: lo0
flags: <UP,HOST,DONE,STATIC,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 16384 1 0
root@OPNsense:~ # route show 10.1.2.3
route to: 10.1.2.3
destination: 10.2.0.0
mask: 255.255.0.0
fib: 0
interface: wg1
flags: <UP,DONE,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1420 1 0
It worked before the 23.7.4 update, soo... waiting for "Note that the WireGuard plugin improvement effort is still going on ..."
Let me know if you need some tests or better diagnostics.
The "pinned" seems strange to me. Can you grep from the routing table to see the flags?
# netstat -nr | grep 10.1.2.3
Something creates the route so that's why the error appears which is a local issue. That's all I can say.
Cheers,
Franco
Thank you, Franco.
The errors there are when I ping 10.1.2.1 from WireGuard client 10.1.2.3:
root@OPNsense:~ # netstat -nr 10.1.2.3
input (Total) output
packets errs idrops bytes packets errs bytes colls
13 5 0 1662 9 0 920 0
10 9 0 1636 1 0 178 0
16 10 0 2356 7 0 688 0
The PINNED flag is there either I delete (unassign) the opt1 from wg1 or not.
Is this maybe because you are using a gateway monitor on the assigned WireGuard interface?
Cheers,
Franco
I can confirm this started happening for me since the upgrade. Yes, I have a gateway monitoring on the assigned Wireguard interfaces that are affected.
As this was working fine before the upgrade, is this considered a 'bug' and being tracked somewhere? Or is this no longer a supported arrangement? Do I need to reconfigure something on my end?
I would really like to keep the monitoring enabled as before if possible.
--
Ross
I'm getting the same error. No gateway monitor, no static routes. After 23.7.5 boot:
2023-09-27T22:17:02 Notice wireguard Wireguard interface WGSite2Site (wg2) started
2023-09-27T22:17:02 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.101.1.2/30' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) can not reconfigure without stopping it first.
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) started
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) can not reconfigure without stopping it first.
> I can confirm this started happening for me since the upgrade. Yes, I have a gateway monitoring on the assigned Wireguard interfaces that are affected.
"Disable Host Route" under System: Gateways: Single gateway setting should fix the issue then.
> I'm getting the same error. No gateway monitor, no static routes.
I think at this point it appears that host routes are the problem. They may also be set up by global DNS servers or other automatic mechanisms.
Cheers,
Franco
I have "Disable Host Route" under System >> Gateways, but after a reboot same Error is show.
2023-10-12T15:28:00 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add '-4' '10.18.0.1' -iface 'wg2'' returned exit code '1', the output was ''
2023-10-12T15:28:00 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add '-4' '10.20.0.1' -iface 'wg1'' returned exit code '1', the output was ''
What's the output of the following command?
# pluginctl -r host_routes
Cheers,
Franco
I've sent you the output via PM.
I've also tried to delete everything. tunnels, interfaces, gateway and firewall rules, and I've created everything again, but after the reboot, the same error shows up.
For the records, everything is working fine. dpinger is working, the gateway is up.... but for some reason I don't know, that's displayed on the Logs.
Quote from: furfix on October 12, 2023, 03:31:09 PM
I have "Disable Host Route" under System >> Gateways, but after a reboot same Error is show.
2023-10-12T15:28:00 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add '-4' '10.18.0.1' -iface 'wg2'' returned exit code '1', the output was ''
2023-10-12T15:28:00 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add '-4' '10.20.0.1' -iface 'wg1'' returned exit code '1', the output was ''
VPN -> WireGuard -> Settings -> Instances -> Edit instance: If "Disable routes" is checked, it will cause the above quoted error.
In "advanced mode" of Edit instance, if uncheck "Disable routes" but leave a filled "Gateway" IP address there, it will say "You have to enable Disable Routes option." A workaround: Remove/clear the "Gateway" IP first, uncheck "Disable routes", and Save, then re-open Edit instance to add the "Gateway" IP address back and Save. Now the UI will no longer force you to put a check mark on "Disable routes". This trick WILL eliminate the above quoted error, however, the system actually IGNORES your filled "Gateway" IP. The beautiful Gateway IP address is sitting there for nothing. It looks like the UI maker doesn't want "Unchecking 'Disable routes'" and "Filled 'Gateway' IP address" to coexist, just like he/she doesn't want a cat and a dog coexist in a house. If you already have a cat, then the house doesn't allow you to bring in a dog. But if you already have a dog, the house allows you to bring in a cat? A UI bug to fix?
A lot of problems occurr since OPNsense 23.7.6 no longer allows applying Static IP to any WG tunnel Interface in the "Interfaces" management. If you have created a Gateway based on that WG interface (for monitoring status or for firewall rule use), without changing the WG interface's "IPv4 Configuration Type" to "Static IPv4", you loss the options to put the static "IPv4 address" and select the Gateway you created as the interface's "IPv4 Upstream Gateway". This will cause a series of problems in OPNsense. These are what I found so far:
1. Automatic rules of "Outbound" in "NAT" will not be generated correctly for the WG network. (You will have to create it manually. - Step 10 in https://docs.opnsense.org/manual/how-tos/wireguard-selective-routing.html)
2. Automatic rules of "Floating" will not be generated correctly for the firewall host itself to use the WG Gateway (You will have to create it manually. - Step 9 in https://docs.opnsense.org/manual/how-tos/wireguard-selective-routing.html)
3. In DDNS (ddclient) service, if you select the WG interface to detect external IP (check ip method: any ip-check provider), the firewall is not actually using the WG interface. (For my setup, it uses the WAN.) Therefore it will not grab your correct external IP through the WG VPN provider.
Before OPNsense 23.7.6, when you were able to specify "Static IP" address and select the "Gateway" for the WG interface in "Interfaces" management, none of the mentioned 3 problems existed. The Outbound NAT rules and Floating rules were automatically and perfectly generated by the system, and how the DDNS (ddclient) service utilizing the WG tunnel interface to detect external IP was working fine.
Essentially, without the "Static IP" and assigned "Gateway" for the WG interface, OPNsense is not treating the WG interface correctly, even with WG VPN connected.
Could it be that the undefined way of setting a static IPv4 mode on the wireguard assigned interface is causing this?
Here is a relevant support effort: https://github.com/opnsense/core/issues/6934
Functionally there is nothing wrong with setting the tunnel address in the wireguard setting which ends up as the static IPv4 anyway. The go implementation may have had a number of restrictions in the past that caused this problem to appear and pop up in suboptimal workarounds in tutorials.
Cheers,
Franco
"Could it be that the undefined way of setting a static IPv4 mode on the wireguard assigned interface is causing this?"
Yes, I can confirm that was the root cause for a lot of buggy things related to WG in 23.7.6.
"Functionally there is nothing wrong with setting the tunnel address in the wireguard setting which ends up as the static IPv4 anyway."
Well, it's Yes and No...
Functionally there is nothing wrong with setting the tunnel address in the WireGuard setting for the WireGuard VPN to work normally, but not for some other important features in OPNsense to recognize and utilize the WG assigned Interface correctly.
For example, users can no longer use DDNS (ddclient) service to check their public IPs THROUGH THE WIREGUARD INTERFACE (no matter in "ddclient" or "native" mode). They can still select the WG Interface as the interface using, but the IP-checking traffic is actually going out through the WAN Interface/IP which leads DDNS grabbing the public IPs of the direct Internet connection instead of the public IPs through the WireGuard VPN.
My test shows that if I DISABLE the WG Interface while keeping the WireGuard VPN CONNECTED, then DDNS (ddclient) / OPNsense (native) will start to truly use the WG Interface/IP as the Source (even it's been disabled) to go out to IP-checking Web/URL to figure out its public IPs, and become working correctly. But unfortunately we simply cannot keep the WG Interface disabled all the time, as we have to use/refer it in many Firewall rules.
> Functionally there is nothing wrong with setting the tunnel address in the WireGuard setting for the WireGuard VPN to work normally, but not for some other important features in OPNsense to recognize and utilize the WG assigned Interface correctly.
You can still assign the interface and create a gateway for it, no? You can also set "Dynamic gateway policy" and see if that works for you.
While it's true that NAT was set up automatically in this case, the gateway and the IPv4 mode adjustments are more or less the same amount of work to set up the manual/hybrid NAT which is more straightforward/explicit anyway.
Keep in mind that WireGuard has had a complicated history on FreeBSD with Go version, first kernel version being removed, then ports tree kernel version, then base kernel version again and the plugin has been a community effort by Michael for all these years since 2019 so bringing it into the core like you would expect it from OpenVPN or IPsec is a little bit of work on both sides (user and development) to get to that point. It's been a long road. :)
Last bit is updating the official documentation and we will be all set for 24.1.
Cheers,
Franco
Can someone dumb-down the explanation of the/a work-around for WG site-to-site? WG used to work great until a couple releases ago for me. Now it's not at all...same issues as what is provided in this thread. greatly appreciated!
Site2site between OPNsense and ?
OpnSense to OpnSense
I have 1 site that is the main site and it connects to 3 remote sites.
The main site has been upgraded and since 'broke' all WG VPNs.
In testing on 2 of the remote sites, I'm easily able to make a WG VPNs between them, however I cannot make any WG VPN work on the main site. No handshakes and "Adding wg route fails returned exit code '1'..." error in logs as well....
Do you have the IP and GW set on the WG interface(s) ?
This should get you going. If it works apply the same changes on the other sites before uprading.
https://forum.opnsense.org/index.php?topic=36403.msg177980#msg177980 (https://forum.opnsense.org/index.php?topic=36403.msg177980#msg177980)
I see the same error since upgrade. Before the upgrade I just added the VPN with routes checkbox and all was fine. Since the update, i had some problems. VPN was up but firewall was blocking specific traffic until i connected to the remote site once (just ssh) and then all was fine ...
Now I've disabled auto routes, created interface and gateway and apart from the annoying log message, all is fine.
Quote from: newsense on October 19, 2023, 03:14:11 AM
Do you have the IP and GW set on the WG interface(s) ?
This should get you going. If it works apply the same changes on the other sites before uprading.
https://forum.opnsense.org/index.php?topic=36403.msg177980#msg177980 (https://forum.opnsense.org/index.php?topic=36403.msg177980#msg177980)
I have WG tunnel address set....I cannot set IP/GW on the interface as it says I cannot assign an IP to a tunnel
Correct, and you can actually assign as many tunnel addresses as you need.
Cheers,
Franco