Packet loss with Wireguard on OPN-2-OPN

[KiX]

Hi Community,

I'm facing a really interesting issue with a Wireguard VPN tunnel.
Setup is "Site DZ" with virtual OPNsense and "Site O" with a hardware firewall with OPNsense, both running latest Business Firmware.

Wireguard "Site O" connects to Wireguard Server on "Site DZ", not vice versa, only one way. Every connection works fine and as expected.
On a routed LAN on "Site DZ", OPNsense is also working as reverse proxy for TLS (tcp/443). One day I thought the connection when browsing one application, which is offloaded by OPNsense, feels a bit strange, I checked ICMP Ping and saw, that I loose every than and now exact 5 Pings, so the connection aborts and I "feel" it when browsing the web applications.
I checked the ping also directly from the OPNsense firewall itself, same packet loss when pinging or MTRing.
When I'm connecting with my computer directly via a second Wireguard instance (Road Warrior), I have no issues with packet loss, so it must be an issue with the second OPNsense firewall - both Wireguard Instances have default MTU.

Today I read about the MSS clamping (https://github.com/opnsense/docs/pull/498) and set it on both firewalls, rebootet, but nothing changed the situation.

Has anyone a idea? Attached is a wireguard dump of the ICMP and Ping, maybe that helps.

Thank you all!

[KiX]

Hi Community,

in the last weeks I tried everything to punch down the issue and today I fixed the issue with luck but want to share some insights from debugging.
I tried a lout about finding the right MTU, but this wasn't the issue. (you can find the best MTU with `tracepath` on Linux)

I read about how to debug wireguard on FreeBSD and set via SSH on OPNsense the debug flag for the NIC (ifconfig wg0 debug), now I saw connection aborts in the wireguard connection itself, so I recognized that I loose every 2min the VPN connection and wg client tries 5sec to reconnect - that was really strange.
I read about a post (forgot the link) where a user told about a issue with UDP itself, so today, because I was out of ideas, I tried to change the default Wireguard Port from 51820 to another high-port and suddenly, I have no more packet loss!
So maybe it was really a issue with UDP on the default wireguard port, maybe a issue with my ISP, I dont know, but now it works.