OpenVPN problems TCP/SSH sessions hang

[rvalle]

HI!

I am having problems with my OpenVPN tunnel, TCP connections are getting stuck.

I use SSH over the tunnel and sessions will become unresponsive.

However, ICMP during the session works flawlessly.

I know that OPNSense is to blame because I suffer the problem with different frequency/intensity. When the issue is very frequent I can restart my OPNSense and the problem goes away for some time.

I have no clue what this could be about. It started to happen about a  week ago, and it is very persistent.

I also don't understand how comes that TCP cannot recover itself, at the end of the day this is the protocol that it is supposed to handle network issues to provide a stable session. As said before  by looking at ICMP traffic you could not notice that any problem is going on.

I have now clue how to debug this, or find out which part of my setup is to blame.

Any idea what could be going on? how to debug it?

Rafael

[rvalle]

After looking in depth into this issue, looks like my network was undergoing an attack.

After banning the offending IP ranges the problem went away.

I could not notice as the firewall seemed in normal state, for example: no traffic or cpu peaks, yet, connections were hanging.

[rvalle]

arrrhhh, the problem persists.

It is true that it is much worse under packet load but the problem is still there.

I will share my attempts to resolve in case someone else is interested or can help:

My rule to allow the SSH traffic in on the OpenVPN interface. I notice that once the SSH session is closed "default deny" logs on LAN interface are logged.

So, I added the rule to the LAN, and also as a floating rule on OpenVPN + LAN interfaces. And did not help.

Then I set up the following ssh client option:

Host *
  ServerAliveInterval 45

It does not fix the issue but at least the client SSH side is closed with "Timeout Server not Responding", which is better than just freezing the terminal. (in particular if one does not know the escape sequece to kill the ssh session, which is ENTER ~~ .)

Not sure where to continue looking for solutions....

[rvalle]

I now found it.

It is related to Gateway monitoring. Our gateway often reports packetloss and even outage. It is strange because traffic not directed at it will perform better.

Gateway monitoring is also used to replaced the default Gateway by another, probably to implement backup links, etc.

There is an advanced Firewall option that is called "Kill States", which kills all states when a gateway is determined to be down.

That is what is killing my TCP sessions.

That option can be disabled in the Firewall, but it is also possible to disable gateway monitoring for the given gateway.

Anyway I am also looking to why is our default gateway lazy replying to ICMP anyway, as I would like to implement a 4G backup line too.

I had have a few SSH sessions opened for hours now, over the OpenVPN, and its just so nice to see it working fine.