OPNsense Forum
Archive => 16.7 Legacy Series => Topic started by: woo on October 28, 2016, 03:52:05 pm
-
Hi all,
(this is not directly related to OPNsense code itself, just a service provided by an OPNsense box, but here are people who know OpenVPN and can probably help me, I'm sure..)
I've now got about 50 regular users on my OPNsense OpenVPN concentrator, and I keep getting complaints that connections are dropping out, mostly around the 1 hour mark.
The log always shows the same picture.. a slew of messages "openvpn[78997]: hans/191.19.25.210:63081 TLS Error: local/remote TLS keys are out of sync: [AF_INET]191.19.25.210:63081 [1]", followed by one "openvpn[78997]: hans/191.19.25.210:63081 [hans] Inactivity timeout (--ping-restart), restarting"
All web research I've done points to this message relating to firewall config issues, but then the connection shouldn't even be able to be established in the first place.
To me, it looks like some part of the keepalive packets either can not be sent or do not arrive.. but I failed to find any details of what the keepalive actually consists of, and which firewall rules I might need to permit it.
Also, it does not seem to match up from a time perspective.. my server has "keepalive 10 30" set, which should kill the session much sooner than one hour, if it really was keepalive related.
I've switched users from UDP to TCP connection mode, with no difference. I've played with the numbers in the keepalive settings, also no change. I can't really just sniff packets on all interfaces for hours, hoping to catch the one that makes trouble, either...
I'm running out of ideas how to debug this further.. so if anyone can provide enlightenment, I'd be really grateful.
Regards
~woo
-
Nobody got any idea how I could dig into that issue further?
-
50 concurrent users may cause some load. What hardware are you using? Any crypto off-load in the CPU or otherwise?
Bart...
-
Like Bart said, it could be a hardware related .
we have 25 users behind A10 Firewall with SSD, we notied some CPU loading,
-
The behaviour is the same, whether it's 3 people logged in, or 50.
CPU load is below 20%, using crypto offloading on a current-gen Xeon.
I'm pretty sure that some handshake packets are dropped somewhere, but I don't know where, or how to sniff it out without digging through all crypted packets..
-
You either have a very beefy piece of hardware to use a Xeon, or you are running OPNsense as a VM. Do you have more platform details please? There are some hypervisor/NIC model/NIC driver combos that have issues with OPNsense and its underlying FreeBSD OS.
Bart...
-
yeah, the OPNsense is currently the only VM on our new ESXi 5.5 host. I'm using the Intel E1000 emulated network device, via the 'em' driver, which is what VMware recommends for FreeBSD.
Generally, networking works fine on that box.. no troubles with throughput or packet loss or anything at all, just these weird VPN disconnects.
-
Is your host up to 5.5 U3? Have you tried vmxnet3 (if_vmx in FreeBSD) instead? Are you using the official VMware tools, or open-vm-tools?
You could also try VMDirectPath I/O for the WAN connection, if the host has some spare NICs.
Bart...
-
My host is 5.5 on most current patch level. I'm using whatever vmtools came with the OPNsense iso, which looks like the official ones. Not much a fan of switching interface type now.. I'm semi in production with that box already, and that idea smells of downtime.
-
Yes, I agree that you need to consider downtime to swap interfaces. Not an awful lot you can do safely while in production without having a fail-over firewall, either through CARP or secondary routing by your clients.
Any mileage in creating a pre-production environment?
Bart...
-
I've now switched the e1000 card for a vmxnet card, but I don't see any difference. Will keep an eye on it for the next few days..
-
No change.. still getting the same errors with the vmxnet as well.
(and I drowned in other projects for the last few weeks, so couldn't investigate this any further).
I'm still having the impression that the keepalive packets are getting lost somewhere, triggering the session restart. (which of course has to fail as the OTP has changed in the meantime, so the cached credentials are useless).
I'll create a second server instance without OTP to see whether at least the automatic session restart works around this problem, that'll buy me some time to get at the original cause.
My "keepalive packets lost" feeling is also reinforced by the problem _seeming_ not to occur for users which have the "redirect gateway" option pushed to their client.. or those users just don't complain.
Kinda annoys me having to debug in production... and lacking the time to do that properly.
-
I've now run some statistics on the logs and the reports from my users.. and there's a weird accumulation in certain connection durations. Most users get disconnected either roughly around 33 minutes or 63 minutes..
I don't have any information about the OSes those users run (commonly Windows 7, 8 or 10), but could there be any reasons that TLS sessions expire/fail to rekey after certain times?
-
This may come from using TOTP if you are using it.
-
Interesting, I was just heading to this forum as I'm lost of what else I could do to get a proper working OpenVPN connection.
I am using an i5-6200U, which is usually not at it's limit at all (it can saturate 350mbps over OpenVPN), but I can't get my system to keep the connection up. I have to reconnect to get it working again (it often seems that I am still connected, but in fact it lost it already), which is not a deal breaker, but I wonder why I can't get it to work properly.
I tried a lot of "keepalive" variations and followed a lot of different advice you can find with Google, now I am wondering, am I the only one or not. It seems I am not...
Who gets a stable connection working and with what settings?
-
Mine is rock solid on an admittedly lightly loaded AMD Phenom 9650 quad core with 4GB RAM:
Remote Access (SSL/TLS + User Auth)
Local Database
tun UDP 1194
TLS authentication with static key
Local CA
DH 2048
AES-256-CBC
SHA256
no hardware crypto
cert depth one (client+server)
adaptive compression enabled
redirect gateway
IPv4 and IPv6 tunnel networks
internal DNS and NTP
advanced option: push "route-ipv6 2000::/3"
Clients:
Cyanogen 13.1.5 Android OpenVPN Connect 1.1.17 (build 76)
macOS Sierra Tunnelblick 3.6.6
-
Not stable with:
Remote Access (SSL/TLS + User Auth)
Local Database
tun UDP 443
TLS authentication with static key
Local CA
DH 4096
AES-256-CBC
SHA512
Intel RDRAND engine - RAND
cert depth one (client+server)
Strict User/CN Matching deactivated
Redirect Gateway activated
Concurrent connections empty
Compression: No Preference (if I deactive I can't get a connection established)
Inter-Client communication activated
Duplicate Connections activated
IPv6 is disabled
Dynamic IP activated
Address Pool activated
Topology activated
DNS Default Domain deactivated
DNS Servers defined
Force DNS cache update activated
NTP Servers deactivated
NetBIOS Options deactivated
Client Management Port deactivated
Use common name deactivated
Advanced Configuration: keepalive 150 450
Verbosity level 1
Renegotiate time empty
Client:
OpenVPN for Android 0.6.63 (Arne Schwabe)
-
This may come from using TOTP if you are using it.
yeah, as I wrote two posts further up.. I know that the automatic restart fails due to the OTP. This was clear to me from the beginning, and expected.
I do NOT know, why the connection drops at all, as long as there is active traffic, and the keepalive ping settings are reasonably short (10 seconds in my case). Even less do I know why the connection drops at such regular intervals. My users are working remotely via RDP, so there is always a constant stream of data, since RDP regularly sends "nothing changed" update packets if the screen is idle.
I am trying to find out, whether that's a result of some settings that OPNsense are using for their OpenVPN implementation, or whether I'm lacking certain settings on my clients, or anything that I'm missing which prevents me from actually using OpenVPN@OPNsense in our production environment.
-
Just out of pure chance, I noticed something in the (i) help for the OpenVPN server settings, specifically the Renegotiation Time: "Renegotiate data channel key after n seconds (default=3600).
When using a one time password, be advised that your connection will automatically drop because your password is not valid anymore."
Now if THAT isn't the reason for my dropped connections, I don't know what else is.
This side effect might need being made a little more public, don't you think? It de facto means that key renegotiation and OTP are mutually exclusive, which will certainly be an interesting decision for business users.
-
Were you able to find a solution to this problem? I am experiencing the same problem here.
-
For User Auth type OpenVPN servers there is a setting at the very bottom: Renegotiate time
The clients need to set this setting as well, if you have used the client exporter, you need to reexport after setting this setting to "0" for disabled.
Cheers,
Franco
-
Hi Franco,
Thanks for the reply, I must be blind because I have been in VPN -> Servers -> Edit Server for my (Remote Access (User Auth) Server and that setting does not exist at the very bottom. The last option i see is under Advanced Configuration - Verbosity level 1 (Default). What am I doing wrong?
Please see the attached screenshots showing the top of the page and the bottom of the page in that section. I also tried using Chrome & Firefox thinking perhaps there was a problem rendering this option.
-
Hmm, it is only there for server modes "Remote Access (SSL/TLS + User Auth)" or "Remote Access (User Auth)". Otherwise 2FA/TOTP plays no role, because there is no user/password combo to ask.
If you are using a different server mode the problem may be elsewhere.
-
I am using Remote Access (User Auth) specifically and I am able to authenticate with the one time password using Google Authenticator. So authentication does work using my SSL VPN. My second thought was could I place this command "reneg-sec 0" in the Advanced Configuration Box at the bottom of the Server configuration page?
-
Yes, reneg-sec is the same as the field. Is this not a 17.1 install? Which version? Must be a recent 16.7.x! :)
-
Okay i'll add that command into the advanced box and try it out. Yes this is a 16.7-i386 build on FreeBSD 10.3-Release-p5 and I must say I've implemented a lot of different features on this appliance already including Port Forwarding, IPsec VPNs and everything has been just amazingly great and I am using an Intel E1000 NIC in a Virtualized ESXi 6 Environment.
Note: I don't think I used the setup wizard to create the OpenVPN server. I just added a server manually. Perhaps that makes a difference whether or not the Renegotiate time option displays or not.
Just to follow up: Should the "Renegotiate time" option not appear under the OPENVPN Settings the command "reneg-sec 0" entered manually into the advanced box has corrected the problem. I have now been connected an hr 1/2 without a disconnect. Thanks for the assistance. For anyone just joining this conversation late both the VPN Client software and the router require this setting.
-
Just for the record, I solved the problem by NOT using "keepalive". Now I have a stable connection.
-
I assume that the mtu and mss settings on the WAN-interface have been set correct?
I struggled a long time with dropping and re-establishing connections on the tunnel. After setting the mtu on 1400 and the mss to 1300 for the WAN-interface, the tunnel is rock-stable. Of course the settings for the vpn-server and client have to match also.
stefan