[HOWTO] Configure WAN MTU with VLAN and / or PPPoE

[meyergru]

So, because still some people might have problems understanding what this is all about, despite there is already another good guide about this:

https://forum.opnsense.org/index.php?topic=21207.0


MTU - how do I set it?

First off, you must know that on you LAN, the default MTU size will most likely be 1500 Bytes.
This is not the same as the ethernet frame size of 1518 Bytes, which includes the MAC and frame check sequence.
Also, this is not the maximum data payload size, called MSS (maximum segment size), which is then 1460 Bytes.

The physical reality is shown here: https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/appneta/GA/analyze-results/network-performance-monitoring-delivery/delivery-results/delivery-diagnostics-messages/maximum-transmission-unit.html

Ignore VLANs on your own network for now, I will explain that later on.


Let's focus on MTU only. When a network packet is transmitted over your WAN interface, you can get lucky and it only gets converted to another medium by an NT (optical or DSL). This is often the case when your WAN configuration only has DHCP and nothing more.

However, there can be setups where your WAN interface is over a VLAN. The VLAN tag is 4 bytes in length, and because the interfaces are stacked on one another, you will see that under OpnSense, your WAN interface may have a VLAN name like "igc0_vlan40". Thus, the lower layer interface has to accommodate both the packet payload and the VLAN header. Since the default MTU of your igc0 ethernet interface is 1500 bytes, the VLAN interface's MTU can only accomodate 1496 bytes, and accordingly, the MTU of igc0_vlan40 will be 1496 bytes.

What happens now is that packets destined to a target on the internet may have to be split up, e.g. when they carry a payload of 1500 bytes, into one packet with 1496 bytes and one with 4 bytes.
Because of the inter-frame gaps and roundtrip times, the 4 byte packet is suboptimal and will cause a slowdown. It would be beneficial if your WAN MTU matches your LAN MTU.

There can also be the case that your internet connection uses PPPoE, which incurs an overhead of 8 bytes, giving you only a net payload MTU of 1492 Bytes. The worst case would be that the PPPoE connection goes through a VLAN, so have have a stack of WAN (PPPoE, 8 bytes overhead) over a VLAN (4 bytes overhead) on an ethernet port. In such cases, you only have 1500 -8 - 4 bytes MTU, leaving you at 1488 bytes.

Both the "normal" ethernet setup and the "worst case" setup are depicted in the image I appended.

OpnSense tries to guess the resulting MTUs, but it sometimes fails.

It is essential that your MTU size is not set too high, because some internet sites cannot correctly train their TCP packets to match the maximum available MTU (this mechanism is called patch MTU discovery).
When you try to contact such a site, packets may get lost on the way, such that you effectively cannot reliably work with them.

There is a Linux tool to find out what maximum MTU your WAN will accomplish here. I added a FreeBSD version of the script at the end, it can be run under OpnSense CLI.

You should set your WAN MTU to the value that the tool will tell you can achieve.

However, as I said, ideally, you will want to match your WAN MTU to your LAN MTU and sometimes, this is feasible.
It clearly depends on your hardware and that of your ISP if that works, because you will have to use a non-standard ethernet packet MTU, a kind of "jumbo" frame.
For the worst-case scenario with both PPPoE and VLAN involved, you would theoretically need an MTU of 1512 on the ethernet port, 1508 on the VLAN created on it and then 1500 for PPPoE. This is the third case in the appended image.

But beware:

1. The OpnSense settings here are somewhat "wrong". If you have a WAN over PPPoE over VLAN, you "should" have to set WAN MTU = 1500, pppoe0 = 1508, ethernet port = 1512, but in reality it works for me with these MTUs:

WAN MTU: 1508 (this also sets pppoe0, which you cannot set directly, but really results in "real" 1500 byte MTU on WAN)
ONT MTU: (this means the physical ethernet port): 1512 if you have a VLAN for PPPoE, 1508 if not.

2. Set the above values in the web UI and then reboot - they cannot be changed via UI manipulations, because the order of application seems to be wrong that way.

Afterwards, verify that the achievable MTU over your WAN connection works with the tool I linked above.

P.S.: As promised, about your local VLANs: Theoretically, the calculations above would also apply to local ethernet ports, but most switches can do jumbo frames on their switching fabric. Also, usually, you will fan out those VLANs on ports as untagged anyway. Therefore, OpnSense thinks it is best to set a VLAN's MTU also at 1500 bytes. That usually works, even when you use a trunk port.
Most network hardware is capable of doing jumbo frames (up to 9014 bytes) anyway. So, normally, you do not have to think about the theoretical difference for an untagged or a tagged VLAN and can use 1500 bytes MTU for either of them.

[Patrick M. Hausen]

Also, this is not the maximum data payload size, called MSS (maximum segment segment), which is then 1460 Bytes.

For IPv4 ;-)

There is a tool to find out what maximum MTU your WAN will accomplish here: https://www.baeldung.com/linux/maximum-transmission-unit-mtu-ip

Doesn't work - neither on Mac OS nor on FreeBSD. Even after replacing #!/bin/bash with #!/bin/sh. Looks like "ping -M do" is not supported on my machines. Linux only?

[meyergru]

Yes. IPv6 is much less problematic, as PMTU is part of the standard and the normal MTU is only 1280 bytes, such that at least most error scenarios are avoided.

[cookiemonster]

Also, this is not the maximum data payload size, called MSS (maximum segment segment), which is then 1460 Bytes.

For IPv4 ;-)

There is a tool to find out what maximum MTU your WAN will accomplish here: https://www.baeldung.com/linux/maximum-transmission-unit-mtu-ip

Doesn't work - neither on Mac OS nor on FreeBSD. Even after replacing #!/bin/bash with #!/bin/sh. Looks like "ping -M do" is not supported on my machines. Linux only?

Will not work with /bin/bash but if you have it installed on your machine, like in my case (don't know why it is using a different prefix) is /usr/local/bin/bash, then it will.

Code Select Expand

OPNsense:~ $ pkg info bash
bash-5.2.26_1
Name           : bash
Version        : 5.2.26_1
Installed on   : Tue Aug  6 12:43:19 2024 BST

[meyergru]

That tool was developed for Linux. And yes, you need bash. One could do the same thing by using ping with the correct payload size under other OSes. Also, the ping options differ with each OS.

Therefore, I added a FreeBSD version of the script to the article that runs without bash.

[cookiemonster]

I was left courious and now I know why it is how it is (my bash installation).
In the past I had installed bash myself. Later i removed it but it seems it is a dependency pulled by Zenarmor. Still using a different prefix but hey, no bad thing in itself.

[Bob.Dig]

MTU - how do I set it?

Would be interested in your opinion. My VDSL-Fritzbox configures its LAN with 1492. If I look what Windows does with it:

Code Select Expand

PS C:\Users\Bobby> Get-NetIPInterface "Ethernet 2"| Format-Table -AutoSize

ifIndex InterfaceAlias AddressFamily NlMtu(Bytes) InterfaceMetric Dhcp    ConnectionState PolicyStore
------- -------------- ------------- ------------ --------------- ----    --------------- -----------
6       Ethernet 2     IPv6                  1492              25 Enabled Connected       ActiveStore
6       Ethernet 2     IPv4                  1500              25 Enabled Connected       ActiveStore
So it looks like Windows is using 1500 for IPv4 and 1492 for IPv6. I can see the same if I set an interface in *Sense with 1492. Does it make sense to you? TIA

[meyergru]

Somewhat... As I said, IPv6 has means to detect the real MTU, so the 1492 seems correctly detected. With IPv4, this only can be detected in a TCP session via PMTU discovery, so Windows stays at the default MTU size of 1500. Potentially, you will face the situation that the Fritzbox has to split your LAN packets in order to not exceed the WAN MTU.

You can set the MTU for an interface manually as well via registry settings, BTW. Also, Windows sometimes has strange ways of auto-determining the MTU. For example, my output showed:

Code Select Expand

PS C:\Windows\System32\WindowsPowerShell\v1.0> Get-NetIPInterface "Ethernet 11"| Format-Table -AutoSize

ifIndex InterfaceAlias AddressFamily NlMtu(Bytes) InterfaceMetric Dhcp    ConnectionState PolicyStore
------- -------------- ------------- ------------ --------------- ----    --------------- -----------
5       Ethernet 11    IPv6                  1496              20 Enabled Connected       ActiveStore
5       Ethernet 11    IPv4                  1496              20 Enabled Connected       ActiveStore

Which seems strange, since my LAN MTU is definitely 1500 Bytes. However, I have a special network adapter that can handle 802.1q, which takes 4 bytes. Only after disabling that could I change my Windows 11 MTU size to 1500 bytes. On another machine with a different adapter, the MTU was 1500 from the start.

Under Windows, you can check your maximum MTU by using 28 bytes less than that as size with "ping -l <size> -f 8.8.8.8".

Interestingly enough, there currently seems to exist a bug in 25.1 that sets the MTU too low. Because of this, there will be a new kernel for 25.1.1.

[Mks]

Hi,
thanks for the interesting post.

You mentioned

OpnSense settings here are somewhat "wrong".

. In my case on the WAN Interface PPPoE (without VLAN), the calculated PPP MTU is according to the GUI 1492 (see MTU1.png).
I've tried your script which returns 1492 too, so it seems to match, however if I enter the 1492 as fixed value in the GUI, the calculated value is set to 1484 (see MTU2.png).
Question, what is the correct configuration? Keep the calculated value or set it to the 1492?

Thanks

[meyergru]

As I said, if you want "effective" 1500 MTU on a PPPoE-over-VLAN connection, your settings must be as follows:

1. WAN MTU: 1508. Never mind, this is wrong and only sets the underlying interface. You could even set this manually to 1500 via CLI, but if you do that via the GUI, it will result in effective 1492. That is what I meant with "wrong". The shown and effective MTUs differ.

2. pppoe MTU (under "advanced settings"): 1508, however, I think this is irrelevant, because the real setting comes from the WAN interface.

3. VLAN MTU: you cannot set that via the GUI, unless you create a named interface.

4. physical interface below VLAN MTU: 1512.

And also, reboot after you have set that, do not assume the GUI settings will apply immediately. You can check if your resulting effective MTU is 1500 byte with the script I provided.

When you look at the MTUs with "ifconfig", you will see these MTUs: pppoe 1508, vlan 1508, physical interface: 1512.

[yeraycito]

3. VLAN MTU: you cannot set that via the GUI, unless you create a named interface.

Vlan created through Interfaces: Devices: VLAN

[meyergru]

Not for me. I see that on the pppoe advanced parameters, not for the vlan.