[SOLVED] Adding wg route falis returned exit code '1', the output was ''

[Dab1362]

Hello, upgrading to    OPNsense 23.7.3 has broken routes with wireguard. Connections get established ok, but routing fails with the following errors. My setup has not changed and has been working on prvevious releases. I have configured it according to https://docs.opnsense.org/manual/how-tos/wireguard-client.html From the wireguard log:

Code Select Expand

2023-09-07T22:23:26 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T22:23:26 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T22:23:26 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T22:23:26 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T21:44:01 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T21:44:01 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T21:44:01 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T21:44:01 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T21:43:16 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T21:43:16 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T21:41:18 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T21:41:18 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T21:41:18 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T21:41:18 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T21:00:01 Notice wireguard Wireguard interface PIA (wg3) started
2023-09-07T21:00:01 Notice wireguard Wireguard interface PIA (wg3) stopped
2023-09-07T20:56:48 Notice wireguard Wireguard interface PIA (wg3) started
2023-09-07T20:56:48 Notice wireguard Wireguard interface PIA (wg3) stopped
2023-09-07T20:56:48 Notice wireguard Wireguard interface wgRoadWarriors (wg2) started
2023-09-07T20:56:48 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T20:56:48 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-07T20:56:48 Notice wireguard Wireguard interface wgRoadWarriors (wg2) stopped
2023-09-07T20:56:48 Notice wireguard Wireguard interface wgScarnet (wg1) started
2023-09-07T20:56:48 Notice wireguard Wireguard interface wgScarnet (wg1) stopped

[franco]

When you run this:

# /sbin/route -n add -'inet' '10.7.0.2/24' -interface 'wg2'

It probably says the route already exists?

I'm not sure it's related to your issue, but I can see the problem with "-q" muting the error message (but why?).


Cheers,
Franco

[Dab1362]

Thanks Franco, it does say it already exists:

Code Select Expand

root@OPNsense:~ #  /sbin/route -n add -'inet' '10.7.0.2/24' -interface 'wg2'
add net 10.7.0.2: gateway wg2 fib 0: route already in table


Its not an issue with the script itself ? "/usr/local/opnsense/scripts/Wireguard/wg-service-control.php" (I have not modified it)

[Dab1362]

Thanks Franco, when I remove -q from the script command /sbin/route -n add I get the already exists error:

Code Select Expand

2023-09-08T10:15:34 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -n add -'inet' '10.7.0.3/24' -interface 'wg2'' returned exit code '1', the output was 'add net 10.7.0.3: gateway wg2 fib 0: route already in table'
2023-09-08T10:15:34 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -n add -'inet' '10.7.0.2/24' -interface 'wg2'' returned exit code '1', the output was 'add net 10.7.0.2: gateway wg2 fib 0: route already in table'

[franco]

Ok, now we don't see this on our end. Do you maybe have a static route set for this previously? It would explain why this fails. And I assume this is after a clean reboot on 23.7.3?


Cheers,
Franco

[Dab1362]

Thanks Franco, I managed to resolve it by completely removing the interface and associated server and peer configurations,  rebooting, then recreating the interface and server/peer conf.

[P3]

I have the same problem.

When I run without -q, I get:

Code Select Expand

add net 10.1.2.3: gateway wg1 fib 0: route already in table

I removed the WireGuard twice, even uninstalled the plugin. Didn't help.
However, I don't think running this is necessary as the route is there as it probably got there from the "Tunnel Address" field, which has the CIDER and this is correct:

Code Select Expand

root@OPNsense:~ # route show 10.1.2.1
   route to: OPNsense
destination: OPNsense
        fib: 0
  interface: lo0
      flags: <UP,HOST,DONE,STATIC,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         1         0
root@OPNsense:~ # route show 10.1.2.3
   route to: 10.1.2.3
destination: 10.2.0.0
       mask: 255.255.0.0
        fib: 0
  interface: wg1
      flags: <UP,DONE,PINNED>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1420         1         0


It worked before the 23.7.4 update, soo... waiting for "Note that the WireGuard plugin improvement effort is still going on ..."

Let me know if you need some tests or better diagnostics.

[franco]

The "pinned" seems strange to me. Can you grep from the routing table to see the flags?

# netstat -nr | grep 10.1.2.3

Something creates the route so that's why the error appears which is a local issue. That's all I can say.


Cheers,
Franco

[P3]

Thank you, Franco.

The errors there are when I ping 10.1.2.1 from WireGuard client 10.1.2.3:

Code Select Expand

root@OPNsense:~ # netstat -nr 10.1.2.3
            input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
        13     5     0       1662          9     0        920     0
        10     9     0       1636          1     0        178     0
        16    10     0       2356          7     0        688     0


The PINNED flag is there either I delete (unassign) the opt1 from wg1 or not.

[franco]

Is this maybe because you are using a gateway monitor on the assigned WireGuard interface?


Cheers,
Franco

[rossigee]

I can confirm this started happening for me since the upgrade. Yes, I have a gateway monitoring on the assigned Wireguard interfaces that are affected.

As this was working fine before the upgrade, is this considered a 'bug' and being tracked somewhere? Or is this no longer a supported arrangement? Do I need to reconfigure something on my end?

I would really like to keep the monitoring enabled as before if possible.

--
Ross

[Kinerg]

I'm getting the same error. No gateway monitor, no static routes. After 23.7.5 boot:

Code Select Expand

2023-09-27T22:17:02 Notice wireguard Wireguard interface WGSite2Site (wg2) started
2023-09-27T22:17:02 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add -'inet' '10.101.1.2/30' -interface 'wg2'' returned exit code '1', the output was ''
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGSite2Site (wg2) can not reconfigure without stopping it first.
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) started
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) stopped
2023-09-27T22:17:01 Notice wireguard Wireguard interface WGxInternet (wg1) can not reconfigure without stopping it first.


[franco]

> I can confirm this started happening for me since the upgrade. Yes, I have a gateway monitoring on the assigned Wireguard interfaces that are affected.

"Disable Host Route" under System: Gateways: Single gateway setting should fix the issue then.

> I'm getting the same error. No gateway monitor, no static routes.

I think at this point it appears that host routes are the problem. They may also be set up by global DNS servers or other automatic mechanisms.


Cheers,
Franco

[furfix]

I have "Disable Host Route" under System >> Gateways, but after a reboot same Error is show.

Code Select Expand

2023-10-12T15:28:00 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add '-4' '10.18.0.1' -iface 'wg2'' returned exit code '1', the output was ''
2023-10-12T15:28:00 Error wireguard /usr/local/opnsense/scripts/Wireguard/wg-service-control.php: The command '/sbin/route -q -n add '-4' '10.20.0.1' -iface 'wg1'' returned exit code '1', the output was ''

[franco]

What's the output of the following command?

# pluginctl -r host_routes


Cheers,
Franco