Connection established but not established?

[tadgy]

I have a weird problem since updating to 25.7....

When I try to ssh to one of my servers, it completely hangs - no error message, no timeout; just hangs.
But, if I try the connection again immediately after, it works absolutely fine.  So, the first attempt hangs, the second attempt goes through fine.  This is also affecting other connections to the server, like HTTPS.
It's not just this server - this happens with different servers on completely different networks.

I was banging my head thinking it might be IPv6, DNS or a routing issue, but it doesn't appear to be.
The connection, according to OPNsense is "established", but the servers I'm trying to ssh to doesn't show any connection at all.  OPNsense thinks it's NAT'd the connection and it's established, but the servers have no clue of a connection.

I've attached some pictures that might help.
"phil.png" is my desktop where I'm ssh'ing from.  You can see there are connections established (one being the connection that I needed to have open to get the second "core.png" netstat, and the second being the hung attempt to ssh again from another terminal).
"core.png" is the server I am trying to connect to.  It shows only 1 ssh connection - the one I was using the get the netstat.  The second connection from another terminal just isn't there.
"opnsense.png" shows the 'session' information that OPNsense thinks are established.  I've filtered the data to only show the correct server is being displayed.  OPNsense thinks there are 2 connections established to the server.

I've disabled the firewalls on the servers - no difference.
I've disabled IPv6 on my desktop - no difference.
I've checked DNS servers are contactable and usable from OPNsense.
I've checked the 'outgoing' firewall rules on OPNsense - it is the standard auto-generated rules for NAT.

I really don't get it.  Can anyone offer any suggestions, help, advice, or a large vodka? :)

Thanks.

[BrandyWine]

Why does the pic of OPNsense have all those red redactions?

[pfry]

A couple things:
- Heh. I don't bother obscuring data. My systems are already public. And you missed a spot.
- Apparently "grep -e" operation is "or", as I don't see a "102" in "core.png". I could, of course, be blind.
- The other columns in "Firewall: Diagnostics: Sessions" (packets, bytes, rule) might be helpful... or not. You can read those. I can think of ways to confuse pf, but it would take effort. And, of course, the "since updating to 25.7" is hopefully coincidence. The session holding/hanging while established suggests a wrapper, but it should show up in netstat. I don't know. Check "Interfaces: Diagnostics: ARP Table" to see if a proxy is intercepting packets? But that should either fail consistently with a half-open session or silently work.

Other than that, unless someone has seen a similar issue, it'll probably take a lot more data on your setup to diagnose. I could supply vodka - it doesn't agree with me, and I can't really drink at all, these days.

[tadgy]

Many thanks for the replies and your attention, but I'm a dumbarse :)

It was an MTU issue - I'd set an MTU to 9000 when the switch was configured for a maximum of 1500.

Consider this resolved :)

[pfry]

[...]It was an MTU issue[...]

Ha! Nice. Got me, too - I didn't even think of it.