Odd TLS handshake issue with a few upstream servers

[cwynd]

Hello All,

We've discovered a strange issue related to connecting to one or two (only) upstream web servers. We're running OPNsense 21.7 (details attached) and it's working very well aside from this one thing that's come to light. We have a forward proxy configured to inspect full HTTP plus SNI only for HTTPS, which seems to be working fine, including banking sites etc as far as I've heard.

But a few users reported very slow connections (timeout or 30 seconds+) to one or two specific web sites, and I've been investigating, and I've managed to reproduce what I think is the problem for one particular instance from either my local bash (linux) or from the shell on the OPNsense firewall. Here's a openssl dialog from my local console:

Code: [Select]

$ openssl s_client -connect www.reddit.com:443 -prexit
CONNECTED(00000005)
140277959643584:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:332:

'Connected' is returned fast and then after about 30 secs the error is reported.
Similarly, from the OPNsense shell I get:

Code: [Select]

# openssl s_client -connect www.reddit.com:443 -prexit
1648418660352:error:0200203C:system library:connect:Operation timed out:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:110:
1648418660352:error:2008A067:BIO routines:BIO_connect:connect error:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:111:
1648418660352:error:0200203C:system library:connect:Operation timed out:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:110:
1648418660352:error:2008A067:BIO routines:BIO_connect:connect error:/usr/src/crypto/openssl/crypto/bio/b_sock2.c:111:
connect:errno=60
The error is again reported after a long delay.

This is only perhaps 30% of the requests, and this is the one and only upstream server I have managed to duplicate it for. The rest of the time the openssl handshake returns normally and fast, and lists the certificate, same as it does for all other servers I've tried.
I cannot find anything getting blocked at the firewall that might relate to this.
I've also tried from our infrastructure outside this firewall (AWS) and haven't been able to reproduce the problem.

For now the user workaround is to hit reload a few times, and eventually a request will go through normally.

I can't find much by googling the specific error (and only one other reference in the OPNsense forums), and most of the search hits seem to relate to upstream server misconfiguration, which seems unlikely in this case.
This is not a 'panic stations' issue, the workaround... works, but it does seem very odd, and I'm interested to try and solve it.

Any ideas welcome, thanks!
cwynd

[cwynd]

Bump

Anybody? I'm getting fed up with complaints about waiting minutes for reddit to load.
What more info would help?

[MartB]

Check MTU and MSS.

[cwynd]

Thanks, will do.