Show Posts
1
Virtual private networks / OpenVPN connection issue
« on: June 07, 2023, 12:37:29 pm »
Hello,
We have an issue with OpenVPN connection. Already connected users observes packet loss, while a new user is connection.
Hardware and top output:
Here is trace of Openvpn process, when user start to connect:
Openvpn debug log:
As you can see above, Openvpn process executes /usr/local/opnsense/scripts/openvpn/ovpn_event.py and call fork() twice. During this time (about 15 seconds) already connected to VPN server users get packet loss.
From Windows-based and Macos pc's users have packet loss. Ping from Linux-based user pc to Openvpn gateway (10.110.210.1) while new Openvpn user is connect:
OpenVPN server config:
OpenVPN client config:
I have already try to reissue certificates, configure new Openvpn servers with various settings, create new users for Openvpn, disable firewall rules, but the issue with packet loss is still present.
Please let me know if any additional information required. I would be glad for any ideas about this issue and how resolve it.
Regards
We have an issue with OpenVPN connection. Already connected users observes packet loss, while a new user is connection.
Code: [Select]
OPNsense 23.1.9-amd64
FreeBSD 13.1-RELEASE-p7
OpenSSL 1.1.1t 7 Feb 2023
OpenVPN 2.6.4 amd64-portbld-freebsd13.1 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [MH/RECVDA] [AEAD]
library versions: OpenSSL 1.1.1t 7 Feb 2023, LZO 2.10
Hardware and top output:
Code: [Select]
hw.model: Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)
hw.machine: amd64
hw.ncpu: 10
last pid: 67728; load averages: 0.25, 2.12, 4.02 up 4+21:22:29 10:21:24
65 processes: 1 running, 64 sleeping
CPU: 0.0% user, 0.0% nice, 0.2% system, 0.2% interrupt, 99.6% idle
Mem: 112M Active, 2532M Inact, 2117M Wired, 1404M Buf, 26G Free
Swap: 8192M Total, 8192M Free
Here is trace of Openvpn process, when user start to connect:
Code: [Select]
sendto(3,"<29>1 2023-06-07T09:21:50.719359"...,149,0,NULL,0) = 149 (0x95)
__sysctl("kern.hostname",2,0x7fffffffbd80,0x7fffffff7c68,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:21:50.719665"...,135,0,NULL,0) = 135 (0x87)
__sysctl("kern.hostname",2,0x7fffffffbb10,0x7fffffff79f8,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:21:50.719982"...,147,0,NULL,0) = 147 (0x93)
__sysctl("kern.hostname",2,0x7fffffffb5b0,0x7fffffff7498,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:21:50.720799"...,302,0,NULL,0) = 302 (0x12e)
fork() = 60508 (0xec5c)
wait4(60508,{ EXITED,val=0 },0x0,0x0) = 60508 (0xec5c)
__sysctl("kern.hostname",2,0x7fffffffb5f0,0x7fffffff74d8,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:21:58.818818"...,244,0,NULL,0) = 244 (0xf4)
__sysctl("kern.hostname",2,0x7fffffffb780,0x7fffffff7668,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:21:58.819194"...,237,0,NULL,0) = 237 (0xed)
__sysctl("kern.hostname",2,0x7fffffffb5b0,0x7fffffff7498,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:21:58.819902"...,293,0,NULL,0) = 293 (0x125)
fork() = 19479 (0x4c17)
wait4(19479,{ EXITED,val=0 },0x0,0x0) = 19479 (0x4c17)
__sysctl("kern.hostname",2,0x7fffffffb5f0,0x7fffffff74d8,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:22:07.267119"...,235,0,NULL,0) = 235 (0xeb)
__sysctl("kern.hostname",2,0x7fffffffb780,0x7fffffff7668,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:22:07.267640"...,228,0,NULL,0) = 228 (0xe4)
__sysctl("kern.hostname",2,0x7fffffffbb10,0x7fffffff79f8,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:22:07.268193"...,162,0,NULL,0) = 162 (0xa2)
__sysctl("kern.hostname",2,0x7fffffffbb10,0x7fffffff79f8,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
sendto(3,"<29>1 2023-06-07T09:22:07.268821"...,162,0,NULL,0) = 162 (0xa2)
__sysctl("kern.hostname",2,0x7fffffffbb10,0x7fffffff79f8,0x0,0) = 0 (0x0)
getpid() = 74430 (0x122be)
Openvpn debug log:
Code: [Select]
<29>1 2023-06-07T09:21:50+00:00 our-server.com openvpn_server 74430 - [meta sequenceId="141560"] xxx.xxx.xxx.xxx:21943 SSL state (accept): TLSv1.3 early data
<29>1 2023-06-07T09:21:50+00:00 our-server.com openvpn_server 74430 - [meta sequenceId="141561"] xxx.xxx.xxx.xxx:21943 TLS: executing verify command: /usr/local/opnsense/scripts/openvpn/ovpn_event.py 1 C=US, ST=Some-state, L=Some-city
, O=Our-company, emailAddress=admin@our-company.com, CN=our-internal-ca
<29>1 2023-06-07T09:21:58+00:00 our-server.com openvpn_server 74430 - [meta sequenceId="141562"] xxx.xxx.xxx.xxx:21943 VERIFY SCRIPT OK: depth=1, C=US, ST=Some-state, L=Some-city, O=Our-company, emailAddress=admin@our-company.com, CN=our-internal-ca
<29>1 2023-06-07T09:21:58+00:00 our-server.com openvpn_server 74430 - [meta sequenceId="141563"] xxx.xxx.xxx.xxx:21943 VERIFY OK: depth=1, C=US, ST=Some-state, L=Some-city, O=Our-company, emailAddress=admin@our-server.com, CN=our-internal-ca
<29>1 2023-06-07T09:21:58+00:00 our-server.com openvpn_server 74430 - [meta sequenceId="141564"] xxx.xxx.xxx.xxx:21943 TLS: executing verify command: /usr/local/opnsense/scripts/openvpn/ovpn_event.py 0 C=US, ST=Some-state, L=Some-city
, O=Our-company, emailAddress=admin@our-company.com, CN=user2
<29>1 2023-06-07T09:22:07+00:00 our-server.com openvpn_server 74430 - [meta sequenceId="141565"] xxx.xxx.xxx.xxx:21943 VERIFY SCRIPT OK: depth=0, C=US, ST=Some-state, L=Some-city, O=Our-company, emailAddress=admin@our-company.com, CN=user2
<29>1 2023-06-07T09:22:07+00:00 our-server.com openvpn_server 74430 - [meta sequenceId="141566"] xxx.xxx.xxx.xxx:21943 VERIFY OK: depth=0, C=US, ST=Some-state, L=Some-city, O=Our-company, emailAddress=admin@our-company.com, CN=user2
<29>1 2023-06-07T09:22:07+00:00 our-server.comm openvpn_server 74430 - [meta sequenceId="141567"] 46.211.230.220:21943 SSL state (accept): SSLv3/TLS read client certificate
<29>1 2023-06-07T09:22:07+00:00 our-server.com openvpn_server 74430 - [meta sequenceId="141568"] xxx.xxx.xxx.xxx:21943 SSL state (accept): SSLv3/TLS read certificate verify
As you can see above, Openvpn process executes /usr/local/opnsense/scripts/openvpn/ovpn_event.py and call fork() twice. During this time (about 15 seconds) already connected to VPN server users get packet loss.
From Windows-based and Macos pc's users have packet loss. Ping from Linux-based user pc to Openvpn gateway (10.110.210.1) while new Openvpn user is connect:
Code: [Select]
64 bytes from 10.110.210.1: icmp_seq=17 ttl=64 time=154 ms
64 bytes from 10.110.210.1: icmp_seq=18 ttl=64 time=154 ms
64 bytes from 10.110.210.1: icmp_seq=19 ttl=64 time=154 ms
64 bytes from 10.110.210.1: icmp_seq=20 ttl=64 time=154 ms
64 bytes from 10.110.210.1: icmp_seq=21 ttl=64 time=154 ms
64 bytes from 10.110.210.1: icmp_seq=22 ttl=64 time=154 ms
64 bytes from 10.110.210.1: icmp_seq=23 ttl=64 time=183 ms
64 bytes from 10.110.210.1: icmp_seq=24 ttl=64 time=24941 ms
64 bytes from 10.110.210.1: icmp_seq=25 ttl=64 time=23904 ms
64 bytes from 10.110.210.1: icmp_seq=26 ttl=64 time=22880 ms
64 bytes from 10.110.210.1: icmp_seq=27 ttl=64 time=21856 ms
64 bytes from 10.110.210.1: icmp_seq=28 ttl=64 time=20833 ms
64 bytes from 10.110.210.1: icmp_seq=29 ttl=64 time=19810 ms
64 bytes from 10.110.210.1: icmp_seq=30 ttl=64 time=18787 ms
64 bytes from 10.110.210.1: icmp_seq=31 ttl=64 time=17762 ms
64 bytes from 10.110.210.1: icmp_seq=32 ttl=64 time=16740 ms
64 bytes from 10.110.210.1: icmp_seq=33 ttl=64 time=15716 ms
64 bytes from 10.110.210.1: icmp_seq=34 ttl=64 time=14693 ms
64 bytes from 10.110.210.1: icmp_seq=35 ttl=64 time=13670 ms
64 bytes from 10.110.210.1: icmp_seq=36 ttl=64 time=12646 ms
64 bytes from 10.110.210.1: icmp_seq=37 ttl=64 time=11623 ms
64 bytes from 10.110.210.1: icmp_seq=38 ttl=64 time=10599 ms
64 bytes from 10.110.210.1: icmp_seq=39 ttl=64 time=9576 ms
64 bytes from 10.110.210.1: icmp_seq=40 ttl=64 time=8552 ms
64 bytes from 10.110.210.1: icmp_seq=41 ttl=64 time=7529 ms
64 bytes from 10.110.210.1: icmp_seq=42 ttl=64 time=6505 ms
64 bytes from 10.110.210.1: icmp_seq=43 ttl=64 time=5482 ms
64 bytes from 10.110.210.1: icmp_seq=44 ttl=64 time=4459 ms
64 bytes from 10.110.210.1: icmp_seq=45 ttl=64 time=3436 ms
64 bytes from 10.110.210.1: icmp_seq=46 ttl=64 time=2412 ms
64 bytes from 10.110.210.1: icmp_seq=47 ttl=64 time=1389 ms
64 bytes from 10.110.210.1: icmp_seq=48 ttl=64 time=365 ms
64 bytes from 10.110.210.1: icmp_seq=49 ttl=64 time=157 ms
64 bytes from 10.110.210.1: icmp_seq=50 ttl=64 time=157 ms
64 bytes from 10.110.210.1: icmp_seq=51 ttl=64 time=157 ms
64 bytes from 10.110.210.1: icmp_seq=52 ttl=64 time=156 ms
64 bytes from 10.110.210.1: icmp_seq=53 ttl=64 time=155 ms
64 bytes from 10.110.210.1: icmp_seq=54 ttl=64 time=155 ms
64 bytes from 10.110.210.1: icmp_seq=55 ttl=64 time=156 ms
64 bytes from 10.110.210.1: icmp_seq=56 ttl=64 time=157 ms
OpenVPN server config:
Code: [Select]
dev ovpns
verb 11
dev-type tun
dev-node /dev/tun
writepid /var/run/openvpn_server.pid
script-security 3
daemon openvpn_server
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
auth SHA256
up /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkup
down /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkdown
local xxx.xxx.xxx.xxx
client-connect "/usr/local/opnsense/scripts/openvpn/ovpn_event.py"
tls-server
server 10.110.210.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/
tls-verify "/usr/local/opnsense/scripts/openvpn/ovpn_event.py"
lport 1199
management /var/etc/openvpn/server.sock unix
ca /var/etc/openvpn/server.ca
cert /var/etc/openvpn/server.cert
key /var/etc/openvpn/server.key
dh /usr/local/etc/inc/plugins.inc.d/openvpn/dh.rfc7919
tls-auth /var/etc/openvpn/server.tls-auth 0
topology subnet
OpenVPN client config:
Code: [Select]
dev tun
persist-tun
persist-key
auth SHA256
client
resolv-retry infinite
remote xxx.xxx.xxx.xxx 1199 udp4
lport 0
verify-x509-name "C=US, ST=Some-state, L=Some-city, O=Our-company, emailAddress=admin@our-company.com, CN=our-internal-certificate" subject
remote-cert-tls server
<ca>
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
-----END PRIVATE KEY-----
</key>
<tls-auth>
#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
-----END OpenVPN Static key V1-----
</tls-auth>
key-direction 1
I have already try to reissue certificates, configure new Openvpn servers with various settings, create new users for Openvpn, disable firewall rules, but the issue with packet loss is still present.
Please let me know if any additional information required. I would be glad for any ideas about this issue and how resolve it.
Regards