[SOLVED] unreliable responses through wireguard tunnel, SSL handshake hangs

[statoshi]

Been banging my head for a week trying to figure out what's configured wrong in my new setup. Here's what I've built:

• OPNSense 22.7.11 router that sends all LAN traffic to a WireGuard tunnel
• WireGuard tunnel endpoint is running on a VPS I control
• OPNSense is configured per the Selective Routing Guide
• VPS is configured to forward traffic from the tunnel, firewall is configured to allow traffic

Here's the behavior I'm seeing when I activate the WireGuard tunnel and Gateway in OPNSense:

• I can ping everything successfully
• Traceroute shows that traffic is going out through the WireGuard tunnel
• I'm able to semi-successfully browse the web. "What is my IP" sites show my IP address as the VPS IP address.
• A lot of my web requests simply timeout.
• I'm not seeing any DNS failures

For example:

Code Select Expand


$ wget -d -v https://duckduckgo.com
Setting --verbose (verbose) to 1
DEBUG output created by Wget 1.21.2 on linux-gnu.

URI encoding = 'UTF-8'
Converted file name 'index.html' (UTF-8) -> 'index.html' (UTF-8)
--2023-01-27 11:04:27--  https://duckduckgo.com/
Resolving duckduckgo.com (duckduckgo.com)... 52.149.246.39
Caching duckduckgo.com => 52.149.246.39
Connecting to duckduckgo.com (duckduckgo.com)|52.149.246.39|:443... connected.
Created socket 3.
Releasing 0x000055e7a41d10b0 (new refcount 1).
Initiating SSL handshake.

So, it seems like something is allowing successful egress of my traffic out of the home network and through the tunnel... but sometimes the response traffic is not being correctly routed back to my laptop? Am I missing a rule that isn't in the selective routing guide?

I will note that the failures are repeatable. That is, the requests that fail are always to the same servers. It's not "random" in that sense, but actually predictable.

I saw several posts of folks complaining of similar issues and they usually said they resolved them by lowering the MTU. I've experimented with a wide range of MTUs from 1200 to 1420 and saw no discernible difference.

Further clues: with the tunnel enabled, most sites load on my laptop (but not reddit or duckduckgo) however my phone (when it's connected to wi-fi) has DNS failures and can't reach anything. Though when I enable a separate tunnel client on my phone, it's able to reach the internet just fine.

[zan]

Did you try set the interface MSS?
Based on my experience setting MTU alone is not enough for a tunnel if the tunnel is used to forward internet traffic because some TLS traffic have big payloads and pmtud is often broken.

[statoshi]

Ding ding ding! That fixed it!

For the record, I had removed all MTU settings and was using default for everything after testing many combinations. Setting an MSS value of 1450 immediately fixed my problems with the SSL handshake hanging on many sites. I guess the next question is what the optimal values for MTU and MSS might be...