Lost TCP-packet over Wireguard tunnel in TLS handshake.

[FrejA]

I use OpnSense as a VPN-Gateway for all outgoing traffic to the internet for LAN clients.
I've just recently switched to a single Wireguard Peer as Gateway (previously two OpenVPN-clients as load balanced gateways).

I run OpnSense on bare metal. Supermicro A1SAi, Intel Atom2750F with 4 i354 Nics.

In general everything works great with Wireguard, great latencies, great speeds (>400Mbit/s at 25% load over all cores /w kmod).
But I'm having trouble with a specific website not working over the VPN tunnel.

At first i though '-Hey maybe the exit-node is blocked, in some DDos-filter or something', but that does not seem to be the case here.

The site in question works perfectly without VPN in OpnSense and also with OpenVPN in OpnSense.
the site also works perfectly using the exact same Wireguard-peer connection directly on from a LAN-machine tunneling directly from the client through OpnSense (exact same exit node IP).
But the site can not be loaded if OpnSense is the responsible for running the Wireguard tunnel.

When i run a TCP-Dump it seems that i can actually communicate with the web-server, but I'm missing a specific packet in the TCP-stream when the traffic.
Even with Wireguard up in OpnSense I can telnet to port 80 on the site and get a HTTP-reply back with an actual HTML-response. But I can't establish a proper TLS-session by any means in any browser or using  curl/PowerShell etc.

In the test scenarios with packet dump:
The LAN-client use the same DNS-servers
Underlying IP in the A-record is the same
Wireguard exit node IP is the same
Both machines can reach the server and do get replies back on a packet level.
With OpnSense running Wireguard, the first packet in the Server Hello gets lost on the way to the client,
and is missing from the RAW TCP-dump (All WAN+WG+LAN IFs dumped in OpnSense)

I've tried both with Wireguard-Go and with KMOD, but the result is the exact same.

The stream working vs. broken looks like this (color marked green/red for the packet missing)
 
Working (Wireguard from Lan Client)
TLS   > Client Hello
TCP   < ACK
TLS < Server Hello
TLS   < PSH, ACK (Continuation Data)
TLS   < PSH, ACK (Continuation Data)
TCP   > ACK
TLS   < Certificate Verify finished
And then rest of the handshake and web-data follows just fine

Broken (Wireguard in OpnSense)
TLS   > Client Hello
TCP   < [ACK]
Here the first packet of the TLS Server Hello is always missing in TCP stream
TLS   < [TCP Previous fragment not captured], Continuation Data But I still get continuation data for the missing Server Hello
TCP   > [TCP Dup ACK]
TLS < Continuation Data
TCP   > [TCP Dup ACK]
TCP   > [Keep-Alive]
TLS   < [Keep-Alive, Ack]
And here its just a long string of trying again, trying again, trying again until timeout.

I've Been trying to solve this for a a few days but I can't come up with any reasonable next step in troubleshooting this.
Any bright ideas would be greatly appreciated!

[0x1f6]

I encountered the same problems and after some time figured out that it was a problem with my MTU settings.

Wireguard adds a layer of encapsulation around your packets,
which the wireguard interface accounts for by setting a lower MTU than the default 1500 in the default config.
In my case though, my provider tunnels my IPv4 over IPv6 to save v4 Addresses (DSLite).
This leads to me having a lower than usual MTU, which i need to account for in my wireguard.conf in the interface stanza.

For me
MTU = 1392
seems to work fine.

[FrejA]

...
For me
MTU = 1392
seems to work fine.

Much appreciated suggestion for a solution. That might just be the cause for what I experienced too.
I'm not quite sure, but I may have just blanked out and not even tried adjusting any MTUs and or MSSs values.

Can't test right now, I've long since reverted back to running multiple OpenVPN gateways instead because I also experienced some janky behavior having multiple WireGuard-VPN clients balanced balanced gateways to WAN.

If this solves handshake, and I if i can also fix the janky balancing of WG in OpenSense this would spare me from A LOT of needless complexity and a great deal of hardware resources.

Due to this HTTPS-issue and a few other issues my current plan of action (when my schedule allowed) was to just break out VPN completely from the OpenSenses and set up multiple balanced virtual WG-gateways in DMZ,
load balance LANs through WG-Hosts, the have WG-hosts failover the OpnSense-boxes, each WG-host having a dedicated path. A real bastard solution for a hobbyist with 2x or more Hosts and 4x IFs :S

Many thanks.
If I get around to setting up a new Test-environment (old one is in the grave) I'll report back if this was the solve for me too.