OPNsense Forum

Archive => 23.1 Legacy Series => Topic started by: kpurrucker on March 11, 2023, 02:11:45 am

Title: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: kpurrucker on March 11, 2023, 02:11:45 am
Hey, all.

Today I updated OPNSense from 23.1.1_2->23.1.3. Since then the OpenVPN users can not authenticate with the following message in the OpenVPN log file.

Code: [Select]
2023-03-11T01:52:15+01:00 firewall.name.local openvpn 19578 - [meta sequenceId="44"] user 'username' could not authenticate.
2023-03-11T01:52:15+01:00 firewall.name.local openvpn_server1 66835 - [meta sequenceId="45"] xx.xx.216.156:60711 WARNING: Failed running command (--auth-user-pass-verify): external program exited with error status: 1
2023-03-11T01:52:15+01:00 firewall.name.local openvpn_server1 66835 - [meta sequenceId="46"] xx.xx.216.156:60711 TLS Auth Error: Auth Username/Password verification failed for peer
2023-03-11T01:52:15+01:00 firewall.name.local openvpn_server1 66835 - [meta sequenceId="47"] xx.xx.216.156:60711 [username] Peer Connection Initiated with [AF_INET]xx.xx.216.156:60711

With the Server Mode "Remote Access (SSL/TLS)" instead "Remote Access (SSL/TLS + User Auth)" in OpenVPN Server configuration the login is functional. So the local User Auth Backend seems to be broken.

Has anyone a suggestion?

Thanks much!
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: eugenmayer on March 11, 2023, 10:58:03 am
I have the same issue with the exact same upgrade path.

Since i'am using LDAP as an authentication, it cannot be the local-auth only. Also using Remote Access "User auth". So beside we have the same issue, we have different configurations.

I did tripple check that the LDAP authentication is working under access, also using the test.

Downgraded via
Code: [Select]
opnsense-revert -r 23.1.1_2 opnsense

and everything is working again.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: kpurrucker on March 14, 2023, 11:06:03 am
It seems the bug wasn't reported yet. So I created an issue: https://github.com/opnsense/core/issues/6417
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: AdSchellevis on March 14, 2023, 11:09:13 am
Hi,

In order to debug this, best check how authentication is configured, using a grep:

Code: [Select]
grep -r auth-user-pass-verify /var/etc/openvpn/*.conf

In which case for active servers, it should point to "/usr/local/opnsense/scripts/openvpn/ovpn_event.py”

When it does, it’s also possible to test the script, using a file containing username and password, like:

Code: [Select]
/usr/local/opnsense/scripts/openvpn/ovpn_event.py --script_type user-pass-verify --auth_method via-file --common_name root '1' /tmp/mypass.txt ; echo $?

In which case  /tmp/mypass.txt contains something like:

Code: [Select]
root
opnsense

Best regards,

Ad
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: neiion on March 14, 2023, 01:06:54 pm
Hi
just wanted to post here to say I am also having an openvpn error after running the upgrade. below is a recent log i am getting in openvpn logs. confirmed the same on 2 separate devices



2023-03-14T11:58:05   Error   openvpn_server2   x.x.x.x:42371 TLS Error: TLS handshake failed   
2023-03-14T11:58:05   Error   openvpn_server2   x.x.x.x:42371 TLS Error: TLS object -> incoming plaintext read error   
2023-03-14T11:58:05   Error   openvpn_server2   x.x.x.x:42371 TLS_ERROR: BIO read tls_read_plaintext error   
2023-03-14T11:58:05   Error   openvpn_server2   1x.x.x.x:42371 OpenSSL: error:1417C086:SSL routines:tls_process_client_certificate:certificate verify failed   
2023-03-14T11:58:05   Warning   openvpn_server2   x.x.x.x:42371 WARNING: Failed running command (--tls-verify script): external program exited with error status: 1   
2023-03-14T11:58:05   Warning   openvpn   Certificate depth 2 exceeded max allowed depth of 1.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: neiion on March 14, 2023, 02:39:46 pm
Hi
just wanted to follow up that i got my issue fixed.
below was what fixed my issue.


in openvpn server settings:

i changed certificate depth to 2
 
 then i adjusted the cipher in use

cipher in use was AES-128-CBC

connection is working now for me now after changing
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: malkovich78 on March 15, 2023, 06:08:48 pm
Hi,
Same problem; no luck changing certificate depth to 2.
Same server accepts connections from linux clients, but previously working OpenVPN for Android clients now get this error; no change was made on clients side.

Regards.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: FrankRo on March 15, 2023, 09:33:41 pm
Same problem here with Viscosity 1.10.6b3 (OpenVPN 2.5.9 under the hood) on MacOS 13.2.1. Tried to change cert depth to 2, create new client connection, ... - nothing works - rolled back to 13.1.1_2 - Auth is via Password + TOTP + Client Cert. Site2Site OpenVPN Connection to  22.7.11_1 works without Problems.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: silverspy18 on March 16, 2023, 05:02:45 am
I appear to have the same problem as described in the previous posts as well using a password + TOTP + client certificate. However, I noticed that authentication failed with the error message:

WARNING: Failed running command (--auth-user-pass-verify): external program exited with error status: 1

when using the OpenVPN client option 'static-challenge', but worked when this option was not used and the TOTP was concatenated to the password instead. It would be nice to get the prompt for the TOTP code working again. My current assumption is that there is an issue with the /usr/local/opnsense/scripts/openvpn/user_pass_verify.php script since the 23.1.2 update.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: AdSchellevis on March 16, 2023, 10:29:25 am
@ silverspy18 can you try https://github.com/opnsense/core/commit/b5289522604b7863a5b3bd8c8a5a21a334b1f59c ? This should re-add the static-challenge parsing.


Code: [Select]
opnsense-patch b5289522
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: kpurrucker on March 16, 2023, 02:21:57 pm
@AdSchellevis Thanx! The Patch work on my testsystem.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: malkovich78 on March 16, 2023, 02:35:03 pm
Hi,

Still same problem on my system after applying patch:

2023-03-16T13:04:12   Warning   openvpn_server2   NOTE: the current --script-security setting may allow this configuration to call user-defined scripts      
2023-03-16T13:04:11   Error   openvpn_server2   event_wait : Interrupted system call (code=4)

Regards.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: AdSchellevis on March 16, 2023, 02:38:24 pm
repeating the same message without offering any information I asked for earlier (https://forum.opnsense.org/index.php?topic=32939.msg159704#msg159704) likely isn't going to lead to an improvement. It was sheer luck silverspy18 mentioned static-challenge, otherwise nothing would have changed until now.

Best regards,

Ad
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: maclinuxfree on March 16, 2023, 03:18:09 pm
Same Problem to me. I am using Linux as Client and had to re-export the openvpn config to get it working again.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: franco on March 16, 2023, 05:34:00 pm
Well, full reports would help like Ad suggested. Your issue is probably https://github.com/opnsense/core/commit/4b2b60050

Not sure what OpenVPN is expecting here but we will be reverting to the original (deprecated) behaviour and hope they keep supporting it onwards. ;)


Cheers,
Franco
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: malkovich78 on March 16, 2023, 05:54:09 pm
repeating the same message without offering any information I asked for earlier (https://forum.opnsense.org/index.php?topic=32939.msg159704#msg159704) likely isn't going to lead to an improvement. It was sheer luck silverspy18 mentioned static-challenge, otherwise nothing would have changed until now.

Best regards,

Ad

Tough but fair, I use certifficate-based auth so the following command gived no output:

root@opnsense# grep -r auth-user-pass-verify /var/etc/openvpn/*.conf
root@opnsense#

I attach my edited config and the error logs I get on both sides just in case it helps:

# cat /var/etc/openvpn/server2.conf
dev ovpns2
verb 1
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_server2.pid
script-security 3
daemon openvpn_server2
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
data-ciphers-fallback AES-256-CBC
auth SHA256
up /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkup
down /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkdown
local #edited_ip addr#
client-connect "/usr/local/opnsense/scripts/openvpn/ovpn_event.py '2'"
tls-server
server #network mask#
client-config-dir /var/etc/openvpn-csc/2
ifconfig #edited_ip1 ip2#
tls-verify "/usr/local/opnsense/scripts/openvpn/ovpn_event.py '2'"
lport #edited_port#
management /var/etc/openvpn/server2.sock unix
max-clients 2
push "route #edited_route_1 mask#"
push "route #edited_route_2 mask#"
push "route #edited_route_3 mask#"
push "route #edited_route_4 mask#"
push "route #edited_route_5 mask#"
push "route #edited_route_6 mask#"
ca /var/etc/openvpn/server2.ca
cert /var/etc/openvpn/server2.cert
key /var/etc/openvpn/server2.key
dh /usr/local/etc/inc/plugins.inc.d/openvpn/dh.rfc7919
tls-crypt /var/etc/openvpn/server2.tls-crypt
push "dhcp-option DNS #edited_dns_ip#"
push "dhcp-option DOMAIN #edited_localdomain#"
auth-nocache

Server logs:
2023-03-16T16:51:56   Warning   openvpn_server2   #edited IP#:5033 WARNING: 'keysize' is used inconsistently, local='keysize 256', remote='keysize 128'   
2023-03-16T16:51:56   Warning   openvpn_server2   #edited IP#:5033 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1569', remote='link-mtu 1553'

Client logs:
2023-03-16 16:54:50 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1553', remote='link-mtu 1569'
2023-03-16 16:54:50 WARNING: 'keysize' is used inconsistently, local='keysize 128', remote='keysize 256'
2023-03-16 16:54:50 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
2023-03-16 16:54:50 [#edited IP#] Peer Connection Initiated with [AF_INET]#edited IP#:#edited_port#
2023-03-16 16:54:51 MANAGEMENT: >STATE:1678982091,GET_CONFIG,,,,,,
2023-03-16 16:54:51 SENT CONTROL [#edited IP#]: 'PUSH_REQUEST' (status=1)
2023-03-16 16:54:51 AUTH: Received control message: AUTH_FAILED,Data channel cipher negotiation failed (no shared cipher)
2023-03-16 16:54:51 TCP/UDP: Closing socket

Error logs say clear that there was a cipher negotiation so I changed to AES-256-GCM instead of CBC and now it works, seems that AES-256-CBC is considered deprecated now; previous log messages (Interrupted system call (code=4)) confused me...

Thanks for your help.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: malkovich78 on March 16, 2023, 05:56:15 pm
Well, full reports would help like Ad suggested. Your issue is probably https://github.com/opnsense/core/commit/4b2b60050

Not sure what OpenVPN is expecting here but we will be reverting to the original (deprecated) behaviour and hope they keep supporting it onwards. ;)


Cheers,
Franco

Bingo! that explains why it wasn't taking notice of following line in my config:

data-ciphers-fallback AES-256-CBC

Thanks and regards
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: yourfriendarmando on March 16, 2023, 06:18:07 pm
Hi All

I confirm this broke upgrading from CE 23.1.3 to 23.1.3_4 . They are for my parents house and small non profit. They haven't complained about no internet as they don't use the VPN, but I do for remote checkups.

For 8 clients running on BE, and myself (9), this has not affected any of them.

Is there an easy way to revert? Or will a fix automate rollback to a working version?
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: silverspy18 on March 16, 2023, 11:03:40 pm
@ AdSchellevis, I've confirmed that the patch b5289522 fixed my problem with the client option 'static-challenge' failing. Thanks!
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: yourfriendarmando on March 17, 2023, 04:56:17 am
In my case, I was not able to check because I forgot my laptop, but someone trying to connect, caused a log entry of:

Certificate max depth of 1 exceeded. I always set this value from One (Client + Server) to Two (Client + Intermediate + Server). This allowed the client connection to be successfully established.

I followed instruction in the Manuel about setting up clients, and made a CA, Intermediate CA, and self signed cert. Perhaps the setting of 1 was not actually enforced previously? By setting it to 2, it worked, as I guess that was my own presenting setup of both server and client?

It works, hopefully this gets my issue out of the way in favor of, what I'm sure will be an overhaul, of the interface for OpenVPN 2.6.x series.

Thanks all
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: kpurrucker on March 17, 2023, 10:19:25 pm
The Patch works also on our production system with 100 users. So I can confirm, that the patch solve the problem on systems with Auth via Password + TOTP + Client Cert. Thanks to @AdSchellevis!
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: guest31444 on April 05, 2023, 10:29:43 am
Hello Guys,

is that issue a firmware bug or a missconfig?

when will this problem be solved by an update?

thank you
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: benyamin on April 05, 2023, 11:56:15 pm
@mara, the patch was committed to master three weeks ago.

You might want to update to the latest revision (23.1.5_4) announced here (https://forum.opnsense.org/index.php?topic=33292.0).

You will likely want to consider the patches here (https://forum.opnsense.org/index.php?topic=33314.msg161367#msg161367) too.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: eugenmayer on May 16, 2023, 08:35:36 am
For me, upgrading from 23.1.6 to 23.1.7 broke the OpenVPN authentication again. I have seen that there is a legacy feature maintained for cipher in the patch.

From the logs i would say it is the same / very similar issue.
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: eugenmayer on May 16, 2023, 11:54:32 am
AFAICS this is related to the upgrade to OpenVPN 2.6.3, which is included in 23.1.7 (compared to 2.5.8 in 23.1.6) - the server does crash for us (the entire daemon) when linux clients connect (about 100). 2.5.8 does work just fine combined with 23.1.6

One has to downgrade opnsense to 23.1.6, since it seems like 23.1.7 changed the config for OpenVPN (so it is compatible with 2.6.3?)
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: mimugmail on May 16, 2023, 01:44:36 pm
Without posting any logs its impossible to help :)
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: benyamin on May 16, 2023, 03:47:00 pm
I have seen that there is a legacy feature maintained for cipher in the patch.
AFAICS this is related to the upgrade to OpenVPN 2.6.3, which is included in 23.1.7 (compared to 2.5.8 in 23.1.6) - the server does crash for us (the entire daemon) when linux clients connect (about 100). 2.5.8 does work just fine combined with 23.1.6

One has to downgrade opnsense to 23.1.6, since it seems like 23.1.7 changed the config for OpenVPN (so it is compatible with 2.6.3?)

It's likely because the compatibility behaviour for --cipher present in v2.5 may have been dropped in v2.6. Previously, the algorithm at --cipher was appended to the list at --data-ciphers. Also, the --data-ciphers-fallback option is really only meant to be applicable to v2.3 peers using --enable-small.

You may want to use both the --cipher and the --data-ciphers-fallback options. Depending on compatibility modes it should pick up one of them.

If that doesn't work you can use --data-ciphers AES-256-GCM:AES-128-GCM:?CHACHA20-POLY1305:<the cipher you need> which will retain AEAD options should your peer upgrade from the legacy cipher.

For more info, there's this topic (https://forum.opnsense.org/index.php?topic=27394.0) and this issue (https://github.com/opnsense/core/issues/6420#issuecomment-1471474007).
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: franco on May 16, 2023, 04:41:17 pm
You may want to use both the --cipher and the --data-ciphers-fallback options. Depending on compatibility modes it should pick up one of them.

Ad and me suspected that this might work, but seems to be far from a desired outcome and likely prone to subtle issues depending on how it's being implemented now or in the future.


Cheers,
Franco
Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: eugenmayer on May 16, 2023, 05:38:45 pm
data-ciphers-fallback does work, we tried that. And this was the issue here.
The reason 23.1.3 failed initially was, that cipher was replaced using data-ciphers (only), which will not work for 2.3 OpenVPN clients if those exists (only from 2.4+). And additional issue is, that the default allowed cyphers changed with the server 2.5, blocking AES-128-CBC - and this was the other issue why people with 2.4 clients could potentially not connect with the old 23.1.3 (pre-patched;v.

AFAICS even with 23.1.7 ciphers is still used in the server config - we removed that by using 'none' and using data-ciphers instead in the custom section, with a list of cyphers are clients need (and thus a road to upgrade ciphers) - this allows all our clients to connect and would be the proper fix for the variant introduced 21.1.3 (since as stated, ciphers itself is deprecated and will be removed with 2.7 AFAIR)

The other issue i meanioned, that the VPN server is crashing under 2.6.3 is something new and not related - i just was not aware while I was investigating. It is a new issue and related to 2.6.x upgrade with 23.1.7.

I created https://forum.opnsense.org/index.php?topic=34052.0 to separate the issues.



Title: Re: OpenVPN not work after update OPNSense from 23.1.1_2->23.1.3
Post by: benyamin on May 16, 2023, 09:51:52 pm
You may want to use both the --cipher and the --data-ciphers-fallback options. Depending on compatibility modes it should pick up one of them.
Ad and me suspected that this might work, but seems to be far from a desired outcome and likely prone to subtle issues depending on how it's being implemented now or in the future.
From OpenVPN v2.6.0, --cipher is always ignored in TLS mode and only used for PSK mode and when using --compat-mode version, neither of which are recommended. Clients connecting to remote legacy servers might still need it depending on the server version.

AFAICS even with 23.1.7 ciphers is still used in the server config - we removed that by using 'none' and using data-ciphers instead in the custom section, with a list of cyphers are clients need (and thus a road to upgrade ciphers) - this allows all our clients to connect and would be the proper fix for the variant introduced 21.1.3 (since as stated, ciphers itself is deprecated and will be removed with 2.7 AFAIR)
Presuming you're not doing so already, consider prefixing the --data-ciphers cipher-list with AES-256-GCM:AES-128-GCM:?CHACHA20-POLY1305:. That way as clients progress through your upgrade path they will also be able to make use of an AEAD cipher. Sacrifices of the list's constituent ciphers might be necessary to ensure it remains less than 128 characters in length...