So, because still some people might have problems understanding what this is all about, despite there is already another good guide about this:
https://forum.opnsense.org/index.php?topic=21207.0
MTU - how do I set it?
First off, you must know that on you LAN, the default MTU size will most likely be 1500 Bytes.
This is not the same as the ethernet frame size of 1518 Bytes, which includes the MAC and frame check sequence.
Also, this is not the maximum data payload size, called MSS (maximum segment size), which is then 1460 Bytes.
The physical reality is shown here: https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/appneta/GA/analyze-results/network-performance-monitoring-delivery/delivery-results/delivery-diagnostics-messages/maximum-transmission-unit.html
Ignore VLANs on your own network for now, I will explain that later on.
Let's focus on MTU only. When a network packet is transmitted over your WAN interface, you can get lucky and it only gets converted to another medium by an NT (optical or DSL). This is often the case when your WAN configuration only has DHCP and nothing more.
However, there can be setups where your WAN interface is over a VLAN. The VLAN tag is 4 bytes in length, and because the interfaces are stacked on one another, you will see that under OpnSense, your WAN interface may have a VLAN name like "igc0_vlan40". Thus, the lower layer interface has to accommodate both the packet payload and the VLAN header. Since the default MTU of your igc0 ethernet interface is 1500 bytes, the VLAN interface's MTU can only accomodate 1496 bytes, and accordingly, the MTU of igc0_vlan40 will be 1496 bytes.
What happens now is that packets destined to a target on the internet may have to be split up, e.g. when they carry a payload of 1500 bytes, into one packet with 1496 bytes and one with 4 bytes.
Because of the inter-frame gaps and roundtrip times, the 4 byte packet is suboptimal and will cause a slowdown. It would be beneficial if your WAN MTU matches your LAN MTU.
There can also be the case that your internet connection uses PPPoE, which incurs an overhead of 8 bytes, giving you only a net payload MTU of 1492 Bytes. The worst case would be that the PPPoE connection goes through a VLAN, so have have a stack of WAN (PPPoE, 8 bytes overhead) over a VLAN (4 bytes overhead) on an ethernet port. In such cases, you only have 1500 -8 - 4 bytes MTU, leaving you at 1488 bytes.
Both the "normal" ethernet setup and the "worst case" setup are depicted in the image I appended.
packets.png
OpnSense tries to guess the resulting MTUs, but it sometimes fails.
It is essential that your MTU size is not set too high, because some internet sites cannot correctly train their TCP packets to match the maximum available MTU (this mechanism is called path MTU discovery).
When you try to contact such a site, packets may get lost on the way, such that you effectively cannot reliably work with them.
There is a Linux tool to find out what maximum MTU your WAN will accomplish here (https://www.baeldung.com/linux/maximum-transmission-unit-mtu-ip). I added a FreeBSD version of the script at the end, it can be run under OpnSense CLI.
You should set your WAN MTU to the value that the tool will tell you can achieve.
However, as I said, ideally, you will want to match your WAN MTU to your LAN MTU and sometimes, this is feasible.
It clearly depends on your hardware and that of your ISP if that works, because you will have to use a non-standard ethernet packet MTU, a kind of "jumbo" frame. BTW: A proxmox VirtIO interface (vtnetX) will need configuration for that under Proxmox itself.
For the worst-case scenario with both PPPoE and VLAN involved, you would theoretically need an MTU of 1512 on the ethernet port, 1508 on the VLAN created on it and then 1500 for PPPoE. This is the third case in the appended image.
As an example, if you have a WAN over PPPoE over VLAN, you should have to set WAN MTU = 1500, PPPOE VLAN = 1508, ethernet port = 1512, and it really only works for me with these tightly controlled MTUs:
pppoe0 MTU: 1500 (this must only be set in the advanced settings of pppoe0, not on the WAN interface itself, see screendumps). This will set the WAN MTU.
ONT MTU: (this means the physical ethernet port): 1512 if you have a VLAN for PPPoE, 1508 if not.
PPPOE VLAN MTU: 1508 (if needed in your setup).
2025-03-15 08_27_15-[WAN] _ Interfaces _ OPNsense.mgsoft – Mozilla Firefox.png2025-03-15 08_28_14-Point-to-Point _ Devices _ Interfaces _ OPNsense.mgsoft – Mozilla Firefox.png
To be able to set these values, you will have to assign each of the underlying interfaces a name, which you would normally not need to.
Also, set the above values in the web UI and then reboot, if in doubt - the values sometimes cannot be successfully changed via UI manipulations, because the order of application seems to be wrong that way.
Afterwards, verify that the shown MTUs are as expected via "ifconfig". Also verify that the achievable MTU over your WAN connection works with the tool I linked above. Ideally, you should be able to do get ping replies this command:
"ping -4 -c4 -D -s 1472 1.1.1.1", resulting in an unfragmented 1500 byte packet.
But not to this:
"ping -4 -c4 -D -s 1474 1.1.1.1", resulting in one oversized packet that cannot be transmitted because of the DF bit set.
Usually, you will also get an answer, when you use larger packets, but only when you allow for fragmentation like this:
"ping -4 -c4 -s 1480 1.1.1.1", resulting in two fragmented packets (note the missing "-D" parameter!).
P.S.: As promised, about your local VLANs: Theoretically, the calculations above would also apply to local ethernet ports, but most switches can do jumbo frames on their switching fabric. Also, usually, you will fan out those VLANs on ports as untagged anyway. Therefore, OpnSense thinks it is best to set a VLAN's MTU also at 1500 bytes. That usually works, even when you use a trunk port.
Most network hardware is capable of doing jumbo frames (up to 9014 bytes) anyway. So, normally, you do not have to think about the theoretical difference for an untagged or a tagged VLAN and can use 1500 bytes MTU for either of them.
P.P.S.: If you use Proxmox or other types of virtualisation, you must set the larger MTU of 1512 or 1508 on the underlying host interface(s) as well. For Proxmox, this would be the bridge and ethernet interface for the WAN port for VTNET interfaces or only the ethernet interface with pass-through.
Quote from: meyergru on February 05, 2025, 09:04:11 PMAlso, this is not the maximum data payload size, called MSS (maximum segment segment), which is then 1460 Bytes.
For IPv4 ;-)
Quote from: meyergru on February 05, 2025, 09:04:11 PMThere is a tool to find out what maximum MTU your WAN will accomplish here: https://www.baeldung.com/linux/maximum-transmission-unit-mtu-ip
Doesn't work - neither on Mac OS nor on FreeBSD. Even after replacing #!/bin/bash with #!/bin/sh. Looks like "ping -M do" is not supported on my machines. Linux only?
Yes. IPv6 is much less problematic, as PMTU is part of the standard and the normal MTU is only 1280 bytes, such that at least most error scenarios are avoided.
Quote from: Patrick M. Hausen on February 05, 2025, 09:05:14 PMQuote from: meyergru on February 05, 2025, 09:04:11 PMAlso, this is not the maximum data payload size, called MSS (maximum segment segment), which is then 1460 Bytes.
For IPv4 ;-)
Quote from: meyergru on February 05, 2025, 09:04:11 PMThere is a tool to find out what maximum MTU your WAN will accomplish here: https://www.baeldung.com/linux/maximum-transmission-unit-mtu-ip
Doesn't work - neither on Mac OS nor on FreeBSD. Even after replacing #!/bin/bash with #!/bin/sh. Looks like "ping -M do" is not supported on my machines. Linux only?
Will not work with /bin/bash but if you have it installed on your machine, like in my case (don't know why it is using a different prefix) is /usr/local/bin/bash, then it will.
OPNsense:~ $ pkg info bash
bash-5.2.26_1
Name : bash
Version : 5.2.26_1
Installed on : Tue Aug 6 12:43:19 2024 BST
That tool was developed for Linux. And yes, you need bash. One could do the same thing by using ping with the correct payload size under other OSes. Also, the ping options differ with each OS.
Therefore, I added a FreeBSD version of the script to the article that runs without bash.
I was left courious and now I know why it is how it is (my bash installation).
In the past I had installed bash myself. Later i removed it but it seems it is a dependency pulled by Zenarmor. Still using a different prefix but hey, no bad thing in itself.
Quote from: meyergru on February 05, 2025, 09:04:11 PMMTU - how do I set it?
Would be interested in your opinion. My VDSL-Fritzbox configures its LAN with 1492. If I look what Windows does with it:
PS C:\Users\Bobby> Get-NetIPInterface "Ethernet 2"| Format-Table -AutoSize
ifIndex InterfaceAlias AddressFamily NlMtu(Bytes) InterfaceMetric Dhcp ConnectionState PolicyStore
------- -------------- ------------- ------------ --------------- ---- --------------- -----------
6 Ethernet 2 IPv6 1492 25 Enabled Connected ActiveStore
6 Ethernet 2 IPv4 1500 25 Enabled Connected ActiveStore
So it looks like Windows is using 1500 for IPv4 and 1492 for IPv6. I can see the same if I set an interface in *Sense with 1492. Does it make sense to you? TIA
Somewhat... As I said, IPv6 has means to detect the real MTU, so the 1492 seems correctly detected. With IPv4, this only can be detected in a TCP session via PMTU discovery, so Windows stays at the default MTU size of 1500. Potentially, you will face the situation that the Fritzbox has to split your LAN packets in order to not exceed the WAN MTU.
You can set the MTU for an interface manually as well via registry settings, BTW. Also, Windows sometimes has strange ways of auto-determining the MTU. For example, my output showed:
PS C:\Windows\System32\WindowsPowerShell\v1.0> Get-NetIPInterface "Ethernet 11"| Format-Table -AutoSize
ifIndex InterfaceAlias AddressFamily NlMtu(Bytes) InterfaceMetric Dhcp ConnectionState PolicyStore
------- -------------- ------------- ------------ --------------- ---- --------------- -----------
5 Ethernet 11 IPv6 1496 20 Enabled Connected ActiveStore
5 Ethernet 11 IPv4 1496 20 Enabled Connected ActiveStore
Which seems strange, since my LAN MTU is definitely 1500 Bytes. However, I have a special network adapter that can handle 802.1q, which takes 4 bytes. Only after disabling that could I change my Windows 11 MTU size to 1500 bytes. On another machine with a different adapter, the MTU was 1500 from the start.
Under Windows, you can check your maximum MTU by using 28 bytes less than that as size with "ping -l <size> -f 8.8.8.8".
Interestingly enough, there currently seems to exist a bug in 25.1 that sets the MTU too low (https://github.com/opnsense/src/issues/235). Because of this, there will be a new kernel for 25.1.1.
Hi,
thanks for the interesting post.
You mentioned
QuoteOpnSense settings here are somewhat "wrong".
. In my case on the WAN Interface PPPoE (without VLAN), the calculated PPP MTU is according to the GUI 1492 (see MTU1.png).
I've tried your script which returns 1492 too, so it seems to match, however if I enter the 1492 as fixed value in the GUI, the calculated value is set to 1484 (see MTU2.png).
Question, what is the correct configuration? Keep the calculated value or set it to the 1492?
Thanks
As I said, if you want "effective" 1500 MTU on a PPPoE-over-VLAN connection, your settings must be as follows:
1. WAN MTU: 1508. Never mind, this is wrong and only sets the underlying interface. You could even set this manually to 1500 via CLI, but if you do that via the GUI, it will result in effective 1492. That is what I meant with "wrong". The shown and effective MTUs differ.
2. pppoe MTU (under "advanced settings"): 1508, however, I think this is irrelevant, because the real setting comes from the WAN interface.
3. VLAN MTU: you cannot set that via the GUI, unless you create a named interface.
4. physical interface below VLAN MTU: 1512.
And also, reboot after you have set that, do not assume the GUI settings will apply immediately. You can check if your resulting effective MTU is 1500 byte with the script I provided.
When you look at the MTUs with "ifconfig", you will see these MTUs: pppoe 1508, vlan 1508, physical interface: 1512.
3. VLAN MTU: you cannot set that via the GUI, unless you create a named interface.
Vlan created through Interfaces: Devices: VLAN
Not for me. I see that on the pppoe advanced parameters, not for the vlan.
Thanks for the great tutorial. I've got a MTU-question, because I'm running OPNsense as a VM with Proxmox:
My Modem "Draytek Vigor" is set to MTU 1492 (ISP: German 1&1 DSL). Proxmox WAN-Port is connected via VirtIO-Bridge to OPNsense-VTNet_WAN. OPNsense establishes the connection via PPPoE and assigns VLAN 7. My MTU-configuration under OPNsense is exactly the way you described it.
Do I also have to change the MTU from 1500 to 1512 for the WAN-Bridge and network-card under Proxmox?
Sounds logical, but I can't confirm proposed configuration actually works in current version of Opnsense (25.1.1). I can confirm, the resolution is down the line.
My setup is PPPoE over VLAN, provider (KPN NL) declares that MTU 1500 on PPPoE is supported.
I run Opnsense in a Proxmox VM with bridge for WAN (bridge and it outbound physical interface in Proxmox set to MTU 1512).
The suggested configuration lead to a broken internet connection, DNS (Unbound with DoT) barely resolve anything.
This does not work:
vtnet1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether bc:24:xx:xx:xx:xx
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan01.6: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
options=80000<LINKSTATE>
ether bc:24:xx:xx:xx:xx
groups: vlan
vlan: 6 vlanproto: 802.1q vlanpcp: 0 parent interface: vtnet1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=0
inet XX.XX.XX.XX--> YY.YY.YY.YY netmask 0xffffffff
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
This obviously works:
vtnet1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
ether bc:24:xx:xx:xx:xx
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan01.6: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=80000<LINKSTATE>
ether bc:24:xx:xx:xx:xx
groups: vlan
vlan: 6 vlanproto: 802.1q vlanpcp: 0 parent interface: vtnet1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
description: WAN (wan)
options=0
inet XX.XX.XX.XX--> YY.YY.YY.YY netmask 0xffffffff
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Not very clear why vtnet1 and vlan01.6 have the same MTU
Maybe the instructions do not work
for you. At least for me, they still do for both 25.1.1 and 25.1.2. Potential reasons for that may be:
1. Your first ifconfig shows a different MTU for pppoe0 (1500) than mine (1508) after applying the configuration. Did you follow
all the steps, including a
reboot?
igc3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
description: ONT (opt15)
options=4e0272b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,LRO,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 60:be:b4:xx:xx:xx
inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::62be:b4ff:fexx:xxxx%igc3 prefixlen 64 scopeid 0x4
groups: LOCAL_VLANS
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igc3_vlan40: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
options=4000000<MEXTPG>
ether 60:be:b4:xx:xx:xx
inet6 fe80::62be:b4ff:fexx:xxxx%igc3_vlan40 prefixlen 64 scopeid 0xf
groups: vlan
vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: igc3
media: Ethernet autoselect (2500Base-T <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
description: WAN (wan)
options=0
inet 93.104.yyy.yyy --> 82.135.16.28 netmask 0xffffffff
inet6 fe80::62be:b4ff:fexx:xxxx%pppoe0 prefixlen 64 scopeid 0x13
inet6 2001:a61:576:zzzz:zzzz:zzzz:zzzz prefixlen 64
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
2. With 25.1.1 kernels, there were reported problems with the MTU (although I had none). You could try 25.1.2.
3. Good for you that your ISP claims to be able to do 1500 MTU over PPPoE. But what makes you think that your KVM "hardware" (vtnet1) can do 1512 MTU? It probably does, but maybe you must do something to enable that first on your VM host (https://forum.proxmox.com/threads/enable-mtu-9000-jumbo-frames.34523/). Also, the underlying physical NIC must be able to do that, too:
Quote from: meyergru on February 05, 2025, 09:04:11 PMIt clearly depends on your hardware and that of your ISP if that works, because you will have to use a non-standard ethernet packet MTU, a kind of "jumbo" frame.
BTW: vtnet1 and vlan0.16 showing the same MTU under normal configurations is obviously wrong, because they cannot be both the same size. About as wrong as pppoe0 incorrectly showing 1508 instead of the resulting 1500 after applying my instructions.
Quote from: meyergru on March 01, 2025, 08:48:41 AMKVM "hardware" (vtnet1) can do 1512 MTU?
That was the issue. VM network device (virtio) actually forces MTU 1500 on Proxmox by default. From Proxmox documentation:
QuoteYou can overwrite the MTU setting for each VM network device. The option mtu=1 represents a special case, in which the MTU value will be inherited from the underlying bridge. This option is only available for VirtIO network devices.
My Linux bridge (and underline hardware NIC) set to MTU 1512, so virtio device at Proxmox side now have the same configured.
vtnet1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
options=80008<VLAN_MTU,LINKSTATE>
ether bc:24:xx:xx:xx:xx
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan01.6: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
options=80000<LINKSTATE>
ether bc:24:xx:xx:xx:xx
groups: vlan
vlan: 6 vlanproto: 802.1q vlanpcp: 0 parent interface: vtnet1
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=0
inet XX.XX.XX.XX--> YY.YY.YY.Y netmask 0xffffffff
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
And this finally works as expected. I updated my initial message.
I've tried your settings @meyergru, and they work great at 25.1, but these do not work for me at kernels 25.1.1 / 25.1.2.
I feel there is still a problem with 25.1.1 / 25.1.2. Especially VPN connections fail to establish (Globalprotect).
Reverting back to 25.1 solves all issues.
Same situation here as asychev (PPPoE over VLAN at KPN in Netherlands), but im runing directly on physical hardware (no VM).
What are settings that will work?
They work for me with all kernels so far, as I already wrote. Did you actually verify your maximum MTU with each setup by using the Baeldung tool or with a manual ping with a corresponding size and the DF bit set?
Does your ifconfig show the correct sizes (after a reboot) - i.e. my values, not asychevs? I mean 1508 bytes on pppoe0.
What hardware NIC is your WAN interface? I repeatedly said that it depends on hardware capabilities and following all of the steps. Also, many ISPs do not use the same network hardware brand in all areas.
And first and foremost: Your VPN connections cannot use the MTU of the underlying WAN connection, I think that should be clear? You obviously must subtract the VPN headers or at least use MSS clamping on your VPN interfaces (see step 5a here (https://docs.opnsense.org/manual/how-tos/wireguard-client.html)).
@Meyergru
Thanks for your assistance! Let me try to answer some of your questions:
- no i did not check maximum MTU with each setup. I assumed that your settings (once they worked for 25.1) that they would be the same and work for 25.1.1/25.1.2 aswell...
- Baeldung tool i have not used. Honestly i am unsure how to run that from cli, but will look into it.
- Manual ping see my first response above. Didnt think it would be necessary after upgrade.
- Please find my current (working) values below under 25.1 (from opnsense->system->routes->status):
WAN_physical: 1512
WAN_VLAN: 1508
WAN_PPPoE: 1500 (filled-in 1508 in GUI)
LAN: 1500
- Hardware NICs are all Intel i210 (it is a dec3840 Deciso box)
- VPN/Globalprotect is using MTU 1400 (standard setting). Question that i have is this low enough? According to this this (https://live.paloaltonetworks.com/t5/globalprotect-articles/troubleshooting-globalprotect-mtu-issues/ta-p/384894) Globalprotect (GP) support page IPSec SSL Tunnel overhead can be 125 bytes. Do i need to subtract 125 bytes from 1500 on the WAN_PPPoE or does GP subtract 125 bytes from 1400 and works with a payload of 1275 bytes? Unclear to me.
Please note that i dont have access to the Globalprotect gateway support environment, so i cannot check all the settings over there.
Hope you can help, getting desperate over here ;)
You should try if you internet MTU is 1500 bytes by using "ping -D -s 1472 8.8.8.8" from the OpnSense CLI. If you do not get a response, then your MTU settings did not work, otherwise they do. You can find the real MTU "X" by lowering the value and adding 28 to the last one working.
Depending on the type of tunnel being used, you must reduce the tunnel interface's MTU by the respective overhead. If it is 126 bytes (like with some specimens of SSL tunnels), you would use (at most) X-126 for the tunnel interface MTU.
From their documentation, I think they assume MTU 1500 and use a default value based on it. So, if your WAN MTU is 1500, they should guess correctly, but you can always try lower values anyway. If a value of 1300 does not work, the something else must be wrong.
All of this assumes IPv4, of course, because IPv6 MTUs are always lower.
@meyergru
Did some further testing onder 25.1 and 25.1.2
As said previously, major issues with Globalprotect under 25.1.1/25.1.2, but no issues at all under 25.1.
- ifconfig from opnsense cli: show exactly the same output for both 25.1 and 25.1.1/25.1.2. No change in MTU values reported. Please see dump below for reference.
- ping to quad9 servers from non-globalprotect laptop max out at 1472 (as expected), which translates to an MTU of 1500 on pppoe. Same for both versions.
- ping to quad9 servers from globalprotect laptop max out at 1372, which translates to an MTU of 1400 set for the SSL Tunnel. Same for both versions.
So, i giess back to square one. No discernable differences in settings/attained MTU for all interfaces for both software versions.
Only thing it could be is that these set values are treated differently between 25.1 and 25.1.2 and settings need to be changed.
You mentioned earlier: "Does your ifconfig show the correct sizes (after a reboot) - i.e. my values, not asychevs? I mean 1508 bytes on pppoe0."
As you can see below iconfig reports my MTU on the pppoe (WAN) connection to be 1500 bytes. Should i increase this to 1508?
Unfortunately i cannot try to lower SSL tunnel MTU on globalprotect laptop since this requires admin privileges (which i dont have).
Any other ideas on what can bring improvement?
Are below ifconfig settings correct?
igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: LAN (lan)
options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether f4:90:ea:00:84:0e
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::f690:eaff:fe00:840e%igb0 prefixlen 64 scopeid 0x1
inet6 [redacted] prefixlen 64
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
description: Phys_WAN (opt1)
options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether f4:90:ea:00:84:0f
inet6 fe80::f690:eaff:fe00:840f%igb1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
options=0
groups: enc
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pfsync0: flags=0 metric 0 mtu 1500
options=0
maxupd: 128 defer: off version: 1400
syncok: 1
groups: pfsync
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
options=0
groups: pflog
igb1_vlan6: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
description: vlan (opt2)
options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG>
ether f4:90:ea:00:84:0f
inet6 fe80::f690:eaff:fe00:840f%igb1_vlan6 prefixlen 64 scopeid 0xb
groups: vlan
vlan: 6 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN_pppoe (wan)
options=0
inet [redacted] --> 195.190.228.6 netmask 0xffffffff
inet6 [redacted]%pppoe0 prefixlen 64 scopeid 0xc
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
So it seems that the MTU is not the culprit here, because your WAN MTU obviously can handle 1500 bytes after following this guide.
I understand that your Laptop builds the Globalprotect connection by itself, so there must be something else that goes wrong with 25.1.1 and 25.1.2 kernels. I remember that there were some test kernel versions done by Franco called 25.1.x-nd or something of that sort which healed some problems but have not yet found their way into official versions. You could search the forum to see if any of those kernels fix your problems.
What is unclear to me is that you say that the ping with 1372 bytes works from that Globalprotect laptop but yet you say that the connection itself does not work?
Also, IDK why your and asychev's pppoe0 MTU show 1500 bytes, mine shows 1508 for whatever reason. My ISP also uses a VLAN, so it should be the same technically. But I think that does not matter, since the resulting WAN MTU is 1500 - and that was the main goal.
Please let me clarify my earlier statements.
When running 25.1.1/25.1.2 versions i do not immediately lose connection with globalprotect, but connecting get MUCH harder. Takes a long time to connect and when it does it connects to Beijing, China... After SSLTunnel establishes this is a very slow (unworkable) connection. I gues due to the distance/delays.
Ok, so next step is to try 25.1.x-nd as well as the MTU 1508 bytes pppoe connection setting.
Update: I've installed 25.1.2-nd and im testing it. Looks good sofar with easy GP connections to local GP gateways.
Hi all.
@meyergru I see that kernel 25.1.3 is out.
Can you please confirm that this kernel contains all the fixes/changes that were part of 25.1.2-nd ?
IDK for sure, but I think so, because the only reason to not include the fixes earlier was - from my memory - that 25.1.2 was shortly before release. There were also some other vital IPv6 fixes included in 25.1.3 w/r to ICMPv6 fragmentation issues.
You can always take a snapshot and revert, if there are problems.
My issues were solved by either using kernel 25.1.2-nd or 25.1.3
Hi, i read this entire topic but still have some doubts about how shall I configure my MTUs across the entire stack.
My setup is:
Proxmox with enp1s0 as WAN phisical port.
From here, a VirtIO bridge is created (vmbr2). This is passed to the OPNSense vm as WAN.
Inside OPNSense, i see this as vtnet0. From here, I need to create a VLAN 835 to comply with my ISP requirements (VLAN Ethernet 802.1q).
Finally, a PPPoE connection (RFC 2516) is created.
So, long story short, can anyone clarify which values must be set for
Proxmox side: enp1s0, vmbr2
OPNSense side: vtnet0, vtnet0_vlan835, pppoe0
Atm i have:
| network device | mtu size |
| enp1s0 | 1512 |
| vmbr2 | 1512 |
| vtnet0 | 1512 |
| vtnet0_vlan835 | 1508 |
| pppoe0 | 1500 |
and have specified 1452 as MSS field under Interfaces > WAN.
Ping:
root@OPNsense:~ # ping -c4 -s 1464 -D 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1464 data bytes
1472 bytes from 8.8.8.8: icmp_seq=0 ttl=117 time=3.625 ms
1472 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=4.102 ms
1472 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=4.334 ms
1472 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=3.938 ms
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.625/4.000/4.334/0.258 ms
root@OPNsense:~ #
The whole point of the first posting is to show how to use an MTU of 1500 bytes in order to make packets from the LAN with that "usual" MTU to pass on over the WAN interface without any need for fragmentation. It explains the neccesary steps to do so, even for your specifc case of PPPoE.
The only thing that would be missing here is that you need to set the "physical" NIC MTU as well. WHat I mean by that is that the underlying bridge interface for vtnet0 must be set to 1512 byte MTU as well.
The steps to test if that all succeeds are given as well. You seem to be testing with a smaller payload (i.e. 1464 instead of 1474), so this should always succeed. Also, IDK why you set the MSS, you do not need that. And you give MTU sizes - but what are these? The settings you use to set them or the shown size of ifconfig (as also discussed in the article, there are sometimes differences, at least for pppoe0).
Sorry for the confusion, i'm quite a newbie on the field and trying to understand better but need some guidances :).
What I have pasted is my actual configuration, searching for suggestions as my ISP doesn't publish MTU guidances. My goal is to optimize my network to use the best combination of MTUs across the "different interfaces"..
Please do let me know if any other clarification is required.
IDK qhat the actual question is. If you followed the guide and tested with "ping -D -s 1472", does it work or not?
- If it does and you did it by the letter (plus followed the advice to enlarge the MTU of the Proxmox NIC, too), then fine.
- If it does not work, then obviously you missed something or you ISP cannot handle a jumbo MTU. In the latter case, you have no choice but to reduce your MTU to 1500-8(pppoe headers)-4(VLAN headers) = 1488 bytes. Sometimes, you might succeed by using 1492, because most NICs can handle VLAN tags on top of a 1500 byte MTU natively.
How this works, how to calculate it and the systematic approach is explained in detail in the article, so what is your problem?
Main confusion comes from:
Quoteor the worst-case scenario with both PPPoE and VLAN involved, you would theoretically need an MTU of 1512 on the ethernet port, 1508 on the VLAN created on it and then 1500 for PPPoE. This is the third case in the appended image.
which makes me thinking that both enp1s0 and vmbr1 shall be set on 1512. This would reflect on vtnet0 inside the vm with the same value.
Then I can config via /etc/rc.conf.local (didn't found any other way from the UI) the MTU for the vtnet0_vlan835 to 1508 and finally to configure 1500 in pppoe0. So far, clear.
But then:
Quote1. The OpnSense settings here are somewhat "wrong". If you have a WAN over PPPoE over VLAN, you "should" have to set WAN MTU = 1500, pppoe0 = 1508, ethernet port = 1512
which, to me, is not clear. "WAN MTU" is intended to be the vlan? becase if it's vtnet0 it's in contrast to what said before...
Lastly
QuotePPPOEVLAN MTU: 1508 (if needed in your setup).
again, before, was suggested to set to 1500.
What is not clear? You should have to set pppoe0 to 1500, but this results in a smaller real MTU size. So, to actually generate 1500 on pppoe0, you must set 1508. That is the exact point what I meant when I told you about the difference of the value you set vs. the values you get for PPPoE interfaces. In this case, the distinction is vital.
With a PPPoE connection, your (logical) WAN interface is obviously pppoe0, not any VLAN. Maybe your ISP asks the underlying interface to be on a VLAN or maybe he does not.
It is just as easy as depicted in the article: You want to get MTU 1500 on WAN, which is pppoe0. In order to actually get that, you need to set 1508, just because it does not work as it should with PPPoE.
Then, in order to have the underlying VLAN (if your ISP needs it) accomodate a net 1500 MTU over PPPoE, you have to add 8 PPPoE header bytes to the net MTU of 1500 bytes, thus the VLAN needs 1508 bytes. In this case, you can also set that, because that works as expected, so set value = get value here.
The underlying ethernet interface beneath the VLAN has to accomodate the 4 byte VLAN header on top of that, so you need 1512 bytes in total. So you set 1512 bytes on the NIC, which, in your case, is vtnet0. Obviously, if there is no VLAN, you can skip the 4 bytes and only set 1508 on the underlying physical interface.
And because you have vtnet0 on a bridge, which connects to a physical NIC on your Proxmox host, you will have to set 1512 (or 1508, depending on VLAN or not) bytes on both of those, as well.
Thanks for your patience.
I have set values according to your suggestions:
root@OPNsense:~ # ifconfig vtnet0 | grep mtu
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
root@OPNsense:~ #root@OPNsense:~ # ifconfig vtnet0_vlan835 | grep mtu
vtnet0_vlan835: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
root@OPNsense:~ #root@OPNsense:~ # ifconfig pppoe0 | grep mtu
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
root@OPNsense:~ #
Proxmox side:
[root@pve-02]: ~ # ifconfig enp1s0 | grep mtu
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1512
[root@pve-02]: ~ #[root@pve-02]: ~ # ifconfig vmbr2 | grep mtu
vmbr2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1512
[root@pve-02]: ~ #
Then i tried with the pings from within the firewall:
root@OPNsense:~ # ping -4 -c4 -D -s 1472 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1472 data bytes
1480 bytes from 8.8.8.8: icmp_seq=0 ttl=117 time=5.410 ms
1480 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=5.976 ms
1480 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=4.952 ms
1480 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=5.074 ms
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.952/5.353/5.976/0.397 ms
root@OPNsense:~ # root@OPNsense:~ # ping -4 -c4 -s 1480 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1480 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # My understanding is that the above ping should result differently according to the first post?
root@OPNsense:~ # ping -4 -c4 -D -s 1480 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1480 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ #
No, everything seems right. 1472 bytes payload work (this is 1500 bytes MTU for ping) and 1480 bytes do not, as expected. Even 1474 bytes should return an error when the packet is transmitted without fragmentation.
You can allow fragmentation for the test ping, but I got mixed results as to if that actually fragments the packet or just drops it - newer kernels seem to drop them. You can experiment with MSS clamping for that, but in fact I never had any problems when "ping -D -s 1474" works. I updated the article to reflect that.
Thanks a lot man :) appreciate your patience.
P.S.: I found, that really with the most current of OpnSense (now 25.1.7_4), there is no more difference between the "set" and "get" values any more. I now can set 1500 on the pppoe0 interface and get the same value in ifconfig.
Also, Google (8.8.8.8) does not answer to fragmented packets any more, so you should use 1.1.1.1 (Cloudflare) as target for ping tests.
The first post has been updated to that effect.
Hi,
as a computer networking newbie I struggle to understand why the VLAN tag has to be considered when setting these MTU values. The 802.1Q VLAN tag is an optional part of the Ethernet header. The MTU values in FreeBSD and other operating systems describe the payload of the Ethernet frame, which is independent of the Ethernet header (except for the allowed minimum payload). Hence, to my understanding, if a VLAN is stacked on another interface in FreeBSD it changes the length of the Ethernet header, not the size of the payload. Therefore FreeBSD allows you to set the MTU of a VLAN equal to the MTU of its parent interface (or smaller, of course).
With this reasoning the baby jumbo frames for PPPoE would always have a MTU of 1508 bytes to accomodate for the 8 byte the PPP header, regardless of whether there is an "ISP customer VLAN" stacked on top or not, because the VLAN tag is not part of the payload but part of the encapsulating Ethernet header. The size of the Ethernet frame varies of course by 4 byte, but the payload stays the same.
What am I missing here? Is the VLAN tag somehow entering the PPPoE payload?
PS @meyergru: thanks for being such a prolific writer here! So far I have been a silent reader and your tutorials are very helpful and taught me lot. Keep 'em coming!
At least today, you are effectively correct, yet, the VLAN header must be fit between the destination MAC and the ethernet type fields, which is then followed by the payload. You can see that in the images here: https://en.wikipedia.org/wiki/IEEE_802.1Q. Also, with QinQ (aka "double tagging"), another 4 bytes are added (The result of adding both a VLAN and a PPPoE header was depicted in my image in the original posting).
Essentially, this shows the difference between an untagged and a tagged frame, which is also the cause of many problem when those are mixed on the same interface and why so many 801.q implementations are broken - even on modern switches.
On older NIC designs, which came before the advents of 801.q, the whole frame could only accomodate 1500 bytes. Thus, adding 4 bytes of VLAN header reduced the remaining payload size accordingly. These days, most hardware supports 801.q with 4 more bytes out of the box and FreeBSD apparently does not even account for it.
Thus, by adding 12 bytes in the case of PPPoE + VLAN, you will always be on the safe side, but usually, 1508 bytes will - probably - do fine.
Take normal VLANs as an example - you normally do not subtract 4 bytes of MTU as you should and it works anyways. If you used QinQ, you might see different results, unless your networking equipment supports even larger frames.
Thank you, confusion elininated :-)
Maybe we should start a list of ISPs where baby jumbo frames work.