Messages

Then here I'm seeing a suggestion that the existing "openvpn" file there is misnamed:

https://www.bountysource.com/issues/65768122-carp-events-notifications-can-be-in-separate-file

What direction is this going now?

Are there docs pertinent to this deeper functionality somewhere? Is it specific to OPNsense, or inherited from somewhere upstream?

Looking at 20-openvpn there it looks like two variables are expected, where the 2nd one is either MASTER or BACKUP, presumably passed to each script in this directory just when carp switches status. The first variable gets read as "subsystem," presumably something about the identity of what triggered the carp status change. I also see
https://docs.opnsense.org/development/backend/autorun.html as reference.

Several questions then:

What are the possible (sane) subsystems for this context? Are there known problem cases to beware of here in the context of taking action, or is it generally safe to ignore this value beyond perhaps logging it?

Are scripts in carp run on initial system startup, just if the system starts up as MASTER?

Thanks!

ifconfig -g tun will list all devices in the "tun" group, which looks sufficient. So the code could call that, then take the numbers from those, and go +1 from the highest number returned for the next tunN to use from OpenVPN. Or perhaps it should start with the highest returned, try that, and go up +1 only if that turns out to be already in use.

Can you post a link to the other thread?

https://forum.opnsense.org/index.php?topic=14282.0

The problem comes in at line 546 in openvpn.inc:

    if (!does_interface_exist($devname)) {
        if (!file_exists("/dev/{$devnode}")) {
            mwexecf('/sbin/ifconfig %s create', array($devnode));
        }


The problem is that does_interface_exist looks specifically for tun devices named ovpnsN and ovpncN, so entirely misses that tunN is in use when it's been renamed wgN instead of ovpnsN or ovpncN for server or client. What's needed is a test for the interface's current use that works no matter what the device may have been renamed to after creation. When tun1 has been renamed wg1, as in my case, this code misses it as it assumes that the only consumer of tunN interfaces is OpenVPN.

This is a new thread based on "OpenVPN from Linux client -- also server fails to start with tun1 error".

In summary, WireGuard starts tunnels using tun devices, renaming them to wg0, wg1 ... wgN. OpenVPN also uses tun devices. In a system with existing wg0, wg1 and wg2 devices, the OpenVPN server startup currently tries to use tun1, which is not available, as already in use as wg1. So OpenVPN simply fails.

Presumably this would not result if only a single wgN device were running based on a renamed tun0, and it's doable in WireGuard to have multiple tunnels based on a single wgN device. But the current OPNsense setup requires each remote endpoint have its own tunN (wgN) device on OPNsense, so that's not a viable workaround when multiple WireGuard tunnels are required.

It's possible that starting OpenVPN before WireGuard would avoid this. Haven't tested that. The system where this has become evident is in production, and the WireGuard tunnels are of higher value to us than OpenVPN access. In any case, the current setup is clearly a bug in the OpenVPN startup, which should be smart enough, on failing to use tun1, to increment N until tunN is found available to it.

[Ah ha! I've got it. There are three wgN interfaces on this system for three WireGuard tunnels. WireGuard interfaces are renamed tunN interfaces. So wg0, wg1, and wg2 are taken. And the OpenVPN startup process is not written so as to transparently do the right thing, and establish itself on tun3.]

From system.log:

Sep 23 14:36:49 OPNsenseFL1 opnsense: /status_services.php: The command '/sbin/ifconfig 'tun1' create' returned exit code '1', the output was 'ifconfig: SIOCIFCREATE2: File exists'
Sep 23 14:36:49 OPNsenseFL1 opnsense: /status_services.php: The command '/sbin/ifconfig 'tun1' name 'ovpns1'' returned exit code '1', the output was 'ifconfig: interface tun1 does not exist'
Sep 23 14:36:49 OPNsenseFL1 opnsense: /status_services.php: The command '/sbin/ifconfig 'ovpns1' group openvpn' returned exit code '1', the output was 'ifconfig: interface ovpns1 does not exist'
Sep 23 14:37:01 OPNsenseFL1 opnsense: /status_services.php: OpenVPN server 1 instance start timed out.

So the script is trying to create tun1, then after that fails persisting in trying to rename it, rather roll the number upwards until it finds a tunN which it can succeed in creating (which will work in the present instance with tun3 and above). Yes, there's a nice contradiction in how the interface both exists and does not exist.

Ah ha! I've got it. There are three wgN interfaces on this system for three WireGuard tunnels. WireGuard interfaces are renamed tunN interfaces. So wg0, wg1, and wg2 are taken. And the OpenVPN startup process is not written so as to transparently do the right thing, and establish itself on tun3.

Now, how should this be fixed?

Hi Bart,

The log file previously showed:

  Sep 20 15:43:03 OPNsenseFL1 openvpn[93890]: Cannot open TUN/TAP dev /dev/tun1: Device busy (errno=16)
  Sep 20 15:43:03 OPNsenseFL1 openvpn[93890]: Exiting due to fatal error

That's why I was looking for the device state, and not finding any /dev/tunN devices at all with ifconfig. However in /dev there are tun0 through tun3 files, and for each but tun3:

  root@OPNsenseFL1:/dev # more tun0
  tun0: Device busy

Interestingly, on the second Deciso device, same model, where we've not used OpenVPN at all, there's just tun0 and tun1, and neither returns "Device busy." Back on the problem system, tun3 does now show up in ifconfig's listing. But the other three don't. How do I see what has these devices "busy" when, if they were properly busy, they should be showing up in ifconfig's device list, but they don't? As "ps aux" shows, there is on openvpn process running at all. What's the right command to see what's claiming business with these /dev/tunN devices?

Honestly, I never set any of these addresses as a /24. Always a /32.

Is your theory that setting it as a /24 would block the writing of the config file for the wireguard instance? It wasn't that, but something obviously prevented it from being created until the Save button on the main page -- not those one the "inside" pages -- was pressed. I'll try this again on our current spare system, which we haven't placed into its future DR role yet.

Thanks Franco,

Exactly what I need to know.

[Ah I see now]

I'm getting this in Ubuntu Linux, after starting the openvpn client from a shell:

Password entry required for 'Enter Auth Username:' (PID 12642).
Please enter password with the systemd-tty-ask-password-agent tool!

What seems wrong is that happens even after I've changed OPNsense's OpenVPN server config to be Remote Access (SSL/TLS) -- without "+ User Auth". So why is it asking for user auth? I've got the certificates installed, and the paths added to the .ovpn file so they're found. I'm starting the client with:

/usr/sbin/openvpn --config /etc/openvpn/openvpn_myid.ovpn

which is quite standard and how I start the client for other OpenVPN servers I run.

I'll grant that systemd is an atrocity, and I've no immediate interest in figuring out how to get the systemd-tty-ask-password-agent tool to work. Why is the OPNsense OpenVPN server asking for user auth though? How do I have it merely accept the certificate? The client log was presenting this as:

Fri Sep 20 15:12:13 2019 ERROR: could not read Auth username from stdin
Fri Sep 20 15:12:13 2019 Exiting due to fatal error

Now, I at first had had "+ User Auth" in the setup. Do I need to take extra action to bounce the OPNsense OpenVPN server process?

Ah I see now, the user config file has a line: auth-user-pass. I was assuming authorization method was controlled solely from the server.

A more serious problem: when I do a "ps aux" or "top" on the server, I don't see any openvpn process. And in /var/log/openvpn.log I see:
Sep 20 15:12:53 OPNsenseFL1 openvpn[44056]: Cannot open TUN/TAP dev /dev/tun1: Device busy (errno=16)
Sep 20 15:12:53 OPNsenseFL1 openvpn[44056]: Exiting due to fatal error

follows by many repeats of ^@^@ etc. Not good. An ifconfig shows no device tun1 (or any tun at all -- usually this would start from tun0). So what's with OPNsense that it can't work tun1 here?

root@OPNsenseFL1:/var/log # ifconfig tun0
ifconfig: interface tun0 does not exist
root@OPNsenseFL1:/var/log # ifconfig tun1
ifconfig: interface tun1 does not exist

This is OPNsense as installed on Deciso appliances. How do I add the tun devices that should be there? This isn't how:

oot@OPNsenseFL1:/var/log # ifconfig tun0 create
ifconfig: SIOCIFCREATE2: File exists
root@OPNsenseFL1:/var/log # ifconfig tun1 create
ifconfig: SIOCIFCREATE2: File exists
root@OPNsenseFL1:/var/log # ifconfig tun0
ifconfig: interface tun0 does not exist
root@OPNsenseFL1:/var/log # ifconfig tun1
ifconfig: interface tun1 does not exist


Thanks,
Whit

What is the most correct way to add arbitrary commands to start or stop services, or execute other actions, when there are CARP status changes? I'm used to doing this in Linux using UCARP, so have the general hang of it. But I'm unacquainted with how BSD, and particularly OPNsense, are set up for this. I'm especially interested in doing it so as to coordinate with, but not get clobbered by, configuration actions taken through the OPNsense web UI. There's likely to be a clean and proper way to do this. Has anyone explored it?

Each address is/was a /32 for the wg interfaces.

The problem where I finally got the 3rd tunnel up and it dropped the 2nd in doing so, until I pressed all the "Save" buttons again without editing anything, looks like a separate thing. But since that firewall is in production, I can't go bouncing the tunnels up and down to see how replicable it is. Might have been a one-off event.

Thanks. Then I'm missing some prerequisite. There are no user names displayed at all.

I've been running OpenVPN servers using easy-rsa to generate certs for years. Obviously this is a different scheme. While I was expecting it to be obvious from the web UI, it's time I read the docs on this aspect.