BUG: OpenVPN Client connects on reboot, but GUI says it doesn't!

[saltyzip]

OPNsense 23.7.r_35-amd64
FreeBSD 13.2-RELEASE-p1
OpenSSL 1.1.1u 30 May 2023

I've had a problem for a while, and it's one of the reasons I switched to the opnsense development version in hope it would fix it, but it hasn't, so maybe raising awareness here might get it on the radar of someone in the know.

I have seen one other post in the form, which I will try and dig out with the exact same issue a while back, but obviously didn't end up reaching a root cause and resolution, however it helped me find a manual fudge to get the GUI back aligned, that is until the next reboot.

[The Problem]
I have an OpenVPN Client which is always running on my opnsense router. It's set to Don't pull routes or add/remove routes automatically, I just use firewall rules to push traffic via it based on some specific ports and it works well.

Problem is after a reboot of the router, the GUI doesn't align to what is actually happening.

Firstly the Gateway is showing VPN Client is up
Secondly the Interfaces is showing the VPN Client is down, yet it has an IP allocated.
Thirdly the VPN Connection Status page is showing Client Type VPN with a Status of failed, and subsequent restarts fail too.

The VPN Log File has the following (snippet) which basically means it's already running in my eyes:
2023-07-24T17:30:56   Notice   openvpn_client1   Exiting due to fatal error   
2023-07-24T17:30:56   Error   openvpn_client1   Cannot open TUN/TAP dev /dev/tun1: Device busy (errno=16)   
2023-07-24T17:30:56   Notice   openvpn_client1   TUN/TAP device ovpnc1 exists previously, keep at

If I remote into the backend and run the following command it shows the client is indeed running:
root@OPNsense:~ # ps auxww | grep openvpn
root    47538   0.0  0.1  18068   7316  -  Ss   16:42     0:00.03 /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.conf
root    56049   0.0  0.1  18068   7324  -  Ss   16:42     0:00.03 /usr/local/sbin/openvpn --config /var/etc/openvpn/server3.conf
root    65001   0.0  0.1  18068   7628  -  Ss   16:33     0:00.20 /usr/local/sbin/openvpn --config /var/etc/openvpn/client1.conf

So what I do next is to kill the client1 pid, 65001 in this instance

The gateway for the client VPN then goes to offline
The Interface still shows down but the IP Address disappears
The VPN Connection Status page is showing Client Type VPN with a Status of blank.

Next on the OpenVPN Connection status window I click the play button to start the VPN Client up.

It shows reconnecting initially and then after a refresh changes to connected and all other fields on the screen get populated. Logs all good too.

If I go back to Gateways it still shows VPN Client as offline
If I look at Interfaces it shows it's up and IP Address assigned.

Last step is I now go into the VPN Client Gateway->single, open up the vpn client gateway to edit it, just click save, then apply changes and that brings the gateway online and everything is finally green.

It's a complete faff, and I really hope this issue can get some development time to fix it. It used to work many many many months ago I think, but some update I assume broke it.

Happy to help diagnose this further if you need more information from me.

Thanks

[saltyzip]

This was the previous forum post I mentioned in my above post, and that I used to help find the workaround:
https://forum.opnsense.org/index.php?topic=6376.msg27194#msg27194

Which resulted in this git bug issue being raised, but nothing ever addressed the fix it seems, If I am reading it correctly:
https://github.com/opnsense/core/issues/1931

[Koldnitz]

Saltyzip,

Are you using IPv6?

I noticed that this issue stopped when I turned off IPv6.

I have a similar setup as yours, I have 2 VPNs set up with lists of different servers that fail over (part of a gateway group).

In my case when I boot my internet gateway (Xfinity) goes up and down multiple times due to something with IPv6 and every time this happens OpenVPN does what you have described (launches a new instance) until I run out of the 5 server slots my VPN (AirVPN) provides and it gives me the failure lines in the log. 

At that point it looks like I am logged in on all 5 at once, but I remain logged into the last 2 and everything that depends on the VPNs works but you cannot use the UI to stop them or restart them and the UI shows them being down but with IP addresses

After restarting OPNsense I just manually kill both connections with from the terminal with ps aux, restart them and everything is rock solid.

If you do not use Ipv6 and this is happening my theory goes out the window, when I turned it off a month or 2 ago it fixed the problem, but I am using it again.

This started happening for me sometime after 23.1, and I believe it has to due with the dynamic config pages.

I assume it will be fixed eventually and we are probably edge cases because no one discusses it on the forum.

Cheers,

[saltyzip]

Howdy! Yes, I do use IPV6 via BT Fiber in the UK, have WAN IPv6 Configuration Type set to DHCPv6 and LAN IPV6 set to Track Interface, works well.

Me switching on IPV6, may well have been the trigger to this issue.

I had thought this might have been a more common issue too.

With little forum discussion I did think maybe my config was corrupt in someway, so spent a while analyzing the XML export file, deleting everything from the GUI and then reimporting it, but as you can tell made diddly squat difference.

What is the best way for me to test the theory, should I just set the IPV6 sections to none in LAN and WAN and reboot?

I seem to remember on pfsense at least there was an IPV6 toggle for on or off, can't remember if I've seen that in opnsense?


Update on this, all I had to do was change the WAN IPV6 setting to some other IPv6 setting (not none) and this immediately breaks the GUI reporting on the client VPN, it shows it straight away as down, but it isn't really down. I then need to kill the client1.conf process from the gui to get it back up and running.

[franco]

Have you tried yesterday's RC update?


Cheers,
Franco

[saltyzip]

Hi Franco,
I've just upgraded to latest version and then rebooted:
OPNsense 23.7.r_54-amd64
FreeBSD 13.2-RELEASE-p1
OpenSSL 1.1.1u 30 May 2023

After reboot, some differences (listed below), have included screenshots for reference to show the initial startup position:

1. WAN PPPOE Gateway is now showing down, wasn't before. and Client VPN still showing offline
2. Interfaces is showing everything green, Client VPN is up and IP address assigned, so all good there now.
3. Services looking better, everything green apart from OpenVPN Client is Red.
4. Open VPN Connection status is showing VPN Client as failed, same as before
In addition to the interface, the process command shows it is running however:
root@OPNsense:~ # ps auxww | grep openvpn
root     8529   0.0  0.1  18068   7308  -  Ss   09:17    0:00.01 /usr/local/sbin/openvpn --config /var/etc/openvpn/server3.conf
root     9286   0.0  0.1  18068   7516  -  Ss   09:17    0:00.02 /usr/local/sbin/openvpn --config /var/etc/openvpn/client1.conf
root    99718   0.0  0.1  18068   7312  -  Ss   09:17    0:00.01 /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.conf
root    41806   0.0  0.0  12720   2396  0  S+   09:32    0:00.00 grep openvpn
root@OPNsense:~ #

To manually fix this I then did the following:
1. To get both the PPPOE and Client VPN Gateways showing green, all I did was go into the Client VPN and save settings and apply, and it then immediately turned everything green. So that is better, but still should have been green from the start as everything is working.
2. To try and fix the openvpn connection status I first tried clicking restart, but showed as failed, this is because client is already running, so next went and killed it from the remote shell, which removed failed in connection status to blank. I also checked gateway and interface also then correctly showed client vpn as down and it did.
3. Next I started the client VPN and it connected.
4. Services are all green, Interfaces are all green, unfortunately VPN Client gateway is showing offline and 100% packet loss..
5. Went back into Gateways->Single and opened client vpn, clicked save and then apply changes and it went straight to green, no packet loss.

So still some work to do it seems to get to the bottom of this one. Hope this email helps track the issues down.

Just thought, I'm going to do another reboot, as previous reboot was forced by the upgrade, see if any difference, will update below.

[saltyzip]

Following on from above, after a manual reboot, the Gateway, Interfaces and Services all show just the VPN Client as down, updated pictures attached, so slightly better position.

To make the VPN Client Gateway green, I again just had to open that gateway, save and apply which brought it back.

However that really is a pointless exercise as I still need to kill the client VPN from remote console, and repeat the above exercise to get the Open VPN connection status to report back correctly.

root@OPNsense:~ # ps auxww | grep openvpn
root     3410   0.0  0.1  18068   7484  -  Ss   09:56    0:00.02 /usr/local/sbin/openvpn --config /var/etc/openvpn/client1.conf
root    75150   0.0  0.1  18068   7316  -  Ss   09:56    0:00.01 /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.conf
root    85029   0.0  0.1  18068   7316  -  Ss   09:56    0:00.01 /usr/local/sbin/openvpn --config /var/etc/openvpn/server3.conf
root    67396   0.0  0.0  12720   2396  0  S+   10:08    0:00.00 grep openvpn
root@OPNsense:~ #

I killed 3410, which again shows vpn client gateway/interface/service as down.

Then started the VPN Client again, connected fine.

Only red now is on the Client Gateway again, so usual open it save and apply changes fixes it again.

Slightly better, but no cigar.

[saltyzip]

Do I need to raise some formal bug request on this one, just wondered what the next steps would be?

[Patrick M. Hausen]

Please open an issue on github.