[HOWTO] Configure WAN MTU with VLAN and / or PPPoE

[asychev]

This obviously works:

Code Select Expand

vtnet1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether bc:24:xx:xx:xx:xx
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan01.6: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether bc:24:xx:xx:xx:xx
        groups: vlan
        vlan: 6 vlanproto: 802.1q vlanpcp: 0 parent interface: vtnet1
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        description: WAN (wan)
        options=0
        inet XX.XX.XX.XX--> YY.YY.YY.YY netmask 0xffffffff
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Not very clear why vtnet1 and vlan01.6 have the same MTU

[meyergru]

Maybe the instructions do not work for you. At least for me, they still do for both 25.1.1 and 25.1.2. Potential reasons for that may be:

1. Your first ifconfig shows a different MTU for pppoe0 (1500) than mine (1508) after applying the configuration. Did you follow all the steps, including a reboot?

Code Select Expand

igc3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
        description: ONT (opt15)
        options=4e0272b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,LRO,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 60:be:b4:xx:xx:xx
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::62be:b4ff:fexx:xxxx%igc3 prefixlen 64 scopeid 0x4
        groups: LOCAL_VLANS
        media: Ethernet autoselect (2500Base-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igc3_vlan40: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
        options=4000000<MEXTPG>
        ether 60:be:b4:xx:xx:xx
        inet6 fe80::62be:b4ff:fexx:xxxx%igc3_vlan40 prefixlen 64 scopeid 0xf
        groups: vlan
        vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: igc3
        media: Ethernet autoselect (2500Base-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
        description: WAN (wan)
        options=0
        inet 93.104.yyy.yyy --> 82.135.16.28 netmask 0xffffffff
        inet6 fe80::62be:b4ff:fexx:xxxx%pppoe0 prefixlen 64 scopeid 0x13
        inet6 2001:a61:576:zzzz:zzzz:zzzz:zzzz prefixlen 64
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

2. With 25.1.1 kernels, there were reported problems with the MTU (although I had none). You could try 25.1.2.

3. Good for you that your ISP claims to be able to do 1500 MTU over PPPoE. But what makes you think that your KVM "hardware" (vtnet1) can do 1512 MTU? It probably does, but maybe you must do something to enable that first on your VM host. Also, the underlying physical NIC must be able to do that, too:

It clearly depends on your hardware and that of your ISP if that works, because you will have to use a non-standard ethernet packet MTU, a kind of "jumbo" frame.

BTW: vtnet1 and vlan0.16 showing the same MTU under normal configurations is obviously wrong, because they cannot be both the same size. About as wrong as pppoe0 incorrectly showing 1508 instead of the resulting 1500 after applying my instructions.

[asychev]

KVM "hardware" (vtnet1) can do 1512 MTU?

That was the issue. VM network device (virtio) actually forces MTU 1500 on Proxmox by default. From Proxmox documentation:

You can overwrite the MTU setting for each VM network device. The option mtu=1 represents a special case, in which the MTU value will be inherited from the underlying bridge. This option is only available for VirtIO network devices.

My Linux bridge (and underline hardware NIC) set to MTU 1512, so virtio device at Proxmox side now have the same configured.

Code Select Expand

vtnet1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
        options=80008<VLAN_MTU,LINKSTATE>
        ether bc:24:xx:xx:xx:xx
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan01.6: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
        options=80000<LINKSTATE>
        ether bc:24:xx:xx:xx:xx
        groups: vlan
        vlan: 6 vlanproto: 802.1q vlanpcp: 0 parent interface: vtnet1
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: WAN (wan)
        options=0
        inet XX.XX.XX.XX--> YY.YY.YY.Y netmask 0xffffffff
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

And this finally works as expected. I updated my initial message.

[Kets_One]

I've tried your settings @meyergru, and they work great at 25.1, but these do not work for me at kernels 25.1.1 / 25.1.2.
I feel there is still a problem with 25.1.1 / 25.1.2. Especially VPN connections fail to establish (Globalprotect).
Reverting back to 25.1 solves all issues.

Same situation here as asychev (PPPoE over VLAN at KPN in Netherlands), but im runing directly on physical hardware (no VM).
What are settings that will work?

[meyergru]

They work for me with all kernels so far, as I already wrote. Did you actually verify your maximum MTU with each setup by using the Baeldung tool or with a manual ping with a corresponding size and the DF bit set?

Does your ifconfig show the correct sizes (after a reboot) - i.e. my values, not asychevs? I mean 1508 bytes on pppoe0.

What hardware NIC is your WAN interface? I repeatedly said that it depends on hardware capabilities and following all of the steps. Also, many ISPs do not use the same network hardware brand in all areas.

And first and foremost: Your VPN connections cannot use the MTU of the underlying WAN connection, I think that should be clear? You obviously must subtract the VPN headers or at least use MSS clamping on your VPN interfaces (see step 5a here).

[Kets_One]

@Meyergru
Thanks for your assistance! Let me try to answer some of your questions:
- no i did not check maximum MTU with each setup. I assumed that your settings (once they worked for 25.1) that they would be the same and work for 25.1.1/25.1.2 aswell...
- Baeldung tool i have not used. Honestly i am unsure how to run that from cli, but will look into it.
- Manual ping see my first response above.  Didnt think it would be necessary after upgrade.
- Please find my current (working) values below under 25.1 (from opnsense->system->routes->status):
WAN_physical: 1512
WAN_VLAN: 1508
WAN_PPPoE: 1500 (filled-in 1508 in GUI)
LAN: 1500

- Hardware NICs are all Intel i210 (it is a dec3840 Deciso box)
- VPN/Globalprotect is using MTU 1400 (standard setting). Question that i have is this low enough? According to this this Globalprotect (GP) support page IPSec SSL Tunnel overhead can be 125 bytes. Do i need to subtract 125 bytes from 1500 on the WAN_PPPoE or does GP subtract 125 bytes from 1400 and works with a payload of 1275 bytes? Unclear to me.
Please note that i dont have access to the Globalprotect gateway support environment, so i cannot check all the settings over there.

Hope you can help, getting desperate over here ;)

[meyergru]

You should try if you internet MTU is 1500 bytes by using "ping -D -s 1472 8.8.8.8" from the OpnSense CLI. If you do not get a response, then your MTU settings did not work, otherwise they do. You can find the real MTU "X" by lowering the value and adding 28 to the last one working.

Depending on the type of tunnel being used, you must reduce the tunnel interface's MTU by the respective overhead. If it is 126 bytes (like with some specimens of SSL tunnels), you would use (at most) X-126 for the tunnel interface MTU.

From their documentation, I think they assume MTU 1500 and use a default value based on it. So, if your WAN MTU is 1500, they should guess correctly, but you can always try lower values anyway. If a value of 1300 does not work, the something else must be wrong.

All of this assumes IPv4, of course, because IPv6 MTUs are always lower.

[Kets_One]

@meyergru
Did some further testing onder 25.1 and 25.1.2
As said previously, major issues with Globalprotect under 25.1.1/25.1.2, but no issues at all under 25.1.

- ifconfig from opnsense cli: show exactly the same output for both 25.1 and 25.1.1/25.1.2. No change in MTU values reported. Please see dump below for reference.
- ping to quad9 servers from non-globalprotect laptop max out at 1472 (as expected), which translates to an MTU of 1500 on pppoe. Same for both versions.
- ping to quad9 servers from globalprotect laptop max out at 1372, which translates to an MTU of 1400 set for the SSL Tunnel. Same for both versions.

So, i giess back to square one. No discernable differences in settings/attained MTU for all interfaces for both software versions.
Only thing it could be is that these set values are treated differently between 25.1 and 25.1.2 and settings need to be changed.

You mentioned earlier: "Does your ifconfig show the correct sizes (after a reboot) - i.e. my values, not asychevs? I mean 1508 bytes on pppoe0."
As you can see below iconfig reports my MTU on the pppoe (WAN) connection to be 1500 bytes. Should i increase this to 1508?

Unfortunately i cannot try to lower SSL tunnel MTU on globalprotect laptop since this requires admin privileges (which i dont have).
Any other ideas on what can bring improvement?
Are below ifconfig settings correct?

igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN (lan)
        options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether f4:90:ea:00:84:0e
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::f690:eaff:fe00:840e%igb0 prefixlen 64 scopeid 0x1
        inet6 [redacted] prefixlen 64
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

igb1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
        description: Phys_WAN (opt1)
        options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether f4:90:ea:00:84:0f
        inet6 fe80::f690:eaff:fe00:840f%igb1 prefixlen 64 scopeid 0x2
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

enc0: flags=0 metric 0 mtu 1536
        options=0
        groups: enc
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

pfsync0: flags=0 metric 0 mtu 1500
        options=0
        maxupd: 128 defer: off version: 1400
        syncok: 1
        groups: pfsync

pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33152
        options=0
        groups: pflog

igb1_vlan6: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
        description: vlan (opt2)
        options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG>
        ether f4:90:ea:00:84:0f
        inet6 fe80::f690:eaff:fe00:840f%igb1_vlan6 prefixlen 64 scopeid 0xb
        groups: vlan
        vlan: 6 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: WAN_pppoe (wan)
        options=0
        inet [redacted] --> 195.190.228.6 netmask 0xffffffff
        inet6 [redacted]%pppoe0 prefixlen 64 scopeid 0xc
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

[meyergru]

So it seems that the MTU is not the culprit here, because your WAN MTU obviously can handle 1500 bytes after following this guide.

I understand that your Laptop builds the Globalprotect connection by itself, so there must be something else that goes wrong with 25.1.1 and 25.1.2 kernels. I remember that there were some test kernel versions done by Franco called 25.1.x-nd or something of that sort which healed some problems but have not yet found their way into official versions. You could search the forum to see if any of those kernels fix your problems.

What is unclear to me is that you say that the ping with 1372 bytes works from that Globalprotect laptop but yet you say that the connection itself does not work?

Also, IDK why your and asychev's pppoe0 MTU show 1500 bytes, mine shows 1508 for whatever reason. My ISP also uses a VLAN, so it should be the same technically. But I think that does not matter, since the resulting WAN MTU is 1500 - and that was the main goal.

[Kets_One]

Please let me clarify my earlier statements.
When running 25.1.1/25.1.2 versions i do not immediately lose connection  with globalprotect, but connecting get MUCH harder. Takes a long time to connect and when it does it connects to Beijing, China... After SSLTunnel establishes this is a very slow (unworkable) connection. I gues due to the distance/delays.

Ok, so next step is to try 25.1.x-nd as well as the MTU 1508 bytes pppoe connection setting.
Update: I've installed 25.1.2-nd and im testing it. Looks good sofar with easy GP connections to local GP gateways.