[SOLVED] MTU issue through Wireguard

[knebb]

Hi,

I have a local network and a remote network both connected through Wireguard VPN with two OPNsese boxes.

I can ping and access nearly all of my remote hosts. But not all. On some of them I have issues:

• access to the device from the same network works fine
• ping works fine to these devices from remote network
• connection through https and Firefox warns me about the self-signed certificate the device uses
• then getting connection timeout

I did a packet trace on both OPNSeses an saw this entry multiple times:

Code: [Select]

IPv4, length 610: 192.168.1.254 > 192.168.1.252: ICMP 192.168.9.50 unreachable - need to frag (mtu 1420), length 576
So the OPNSense rejects a packet because it need to be defragmented due to low MTU and the device in question has the "don't fragment" (DF) bit set.
This is the packet which causes the ICMP message:

Code: [Select]

ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 53198, offset 0, flags [DF], proto TCP (6), length 1500)
    192.168.1.252.80 > 192.168.9.50.49272: Flags [.], cksum 0xe438 (correct), seq 1000:2448, ack 744, win 486, options [nop,nop,TS val 6430537 ecr 851618161], length 1448: HTTP, length: 1448
HTTP/1.1 200 OK
So there is an issue with the MTU size. Default MTU size on Wireguard is 1.420 which is too low for the 1.500 packet the device sends.
Now my question: How can I configure it so the packets go through?
And obvisously why is path MTU discovery not working?

Thans fpr ideas/ hints!
/KNEBB

[knebb]

The first interesting thing is the connection to some of the devices is working fine. So it might be the failing devices do not take the ICMP MTU packets into account.
However, I recently migrated from ofSense/OpenVPN to OPNSense/Wireguard and with the previous setup I did not have any issues- even though the MTU there was low through WAN as well.

[Monviech (Cedrik)]

Try changing the MSS as well. Maybe that helps.

Firewall: Settings: Normalization

For me (I use PPPoE) the wireguard MTU of 1412 and MSS of 1352 works.

Just create a rule for "Interface: Wireguard (Group). Leave everything in the rule on any (its the default) and set "Max mss" on 1360 in your case ( for your 1420 wireguard MTU)

If you have PPPoE use the same settings as me.

[knebb]

You made my day!

Even though I am not using PPPoE here, instead FC connection which has 1500 MTU. So I edited the RED interface and set it to 1500 MTU and 1440 MSS.
Remove any MTU settings on the Wiregueard configs.
No change.

Then I went to Firewall Settings Normalisation just as you suggested. Created a rule for Wireguard group and set the max MSS to 1360.

After this it simply worked!

Wow!

Thanks a lot I had never ever found this without help.

Appreciate your hint!

Thanks
/KNEBB
 

[9axqe]

This solved one nagging issue I had with wireguard tunneling as well. Many thanks for this.

The remaining issue I have is that the fragmentation has a performance impact. I understand fragmentation causes the router to do more work, but the opnsense CPU is bored while I'm running a speedtest which produces less than half the speed I should be getting. If I reduce the MTU on the device I'm testing from, I immediately get the proper speed.

I also get that fragmentation will cause some network overhead, but I don't expect it to multiply traffic by two.

So, what am I missing, how is fragmentation causing this slow down?

I'm asking because reducing the MTU on all devices is not possible on my home network. Most devices are mobile devices and MTU seems to be static setting for these, not a per-network setting. And I don't feel like asking everyone in my household to change their MTU on each device...

[Monviech (Cedrik)]

@9axqe
I stopped using wireguard for chasing the highest speed because of those annoying issues. I got back into ipsec and with ikev2 you can get crazy performance with ease. I didn't use any special tweaks and I could get like 500-600mbit/s through a roadwarrior tunnel (OPNsense to NCP Client on Windows 11, SMB file transfer, aes256-sha256-modp2048). I'm sure in the right conditions you could get 1gbit/s or more.

Wireguard - for me - always stopped getting more speed at around 130mbit-150mbit/s, kmod and go implementation. Probably did something wrong, but I couldn't overcome that threshold somehow.

[9axqe]

Thanks @Monviech, this is very valuable experience/information.

I'm trying to VPN my home network to a VPN provider for added security and privacy.

The problem I have is that I have found a single VPN provider so far which is not blocked by my employer. This provider only offers WIreguard.

If you know a good VPN provider which is unlikely to be on any blocklists and supports IPSec, I am interested.

[Monviech (Cedrik)]

Sorry I don't know much about VPN providers. I guess if you want total control you would need some sort of vps with root access somewhere in the cloud and configure your choice VPN into it so its your internet breakout point. But that won't give you the added privacy. In my opinion its only about privacy, not about security.

But employers usually block most things, especially ipsec esp, udp 500 and udp 4500. Some block all UDP ports as well (which would block wireguard). The only thing that most of the time got through was OpenVPN on tcp 443. But even that is blocked with more recent technologies like deep packet inspection and app control.

Some block by dns (which is very easy to circumvent).

So if wireguard works for you, stick with it.

[9axqe]

Ah yes, they block pretty much everything outbound, but I don't mind.

I was talking about inbound connection: my home network connected to a VPN provider, then I want to VPN into my employer's network. They block me if I'm coming from a VPN service they know (I tried expressvpn, mullvad, ivpn, protonvpn, etc.).