[HOWTO] Configure WAN MTU with VLAN and / or PPPoE

[alestark]

Sorry for the confusion, i'm quite a newbie on the field and trying to understand better but need some guidances :).
What I have pasted is my actual configuration, searching for suggestions as my ISP doesn't publish MTU guidances. My goal is to optimize my network to use the best combination of MTUs across the "different interfaces"..
Please do let me know if any other clarification is required.

[meyergru]

IDK qhat the actual question is. If you followed the guide and tested with "ping -D -s 1472", does it work or not?

- If it does and you did it by the letter (plus followed the advice to enlarge the MTU of the Proxmox NIC, too), then fine.

- If it does not work, then obviously you missed something or you ISP cannot handle a jumbo MTU. In the latter case, you have no choice but to reduce your MTU to 1500-8(pppoe headers)-4(VLAN headers) = 1488 bytes. Sometimes, you might succeed by using 1492, because most NICs can handle VLAN tags on top of a 1500 byte MTU natively.

How this works, how to calculate it and the systematic approach is explained in detail in the article, so what is your problem?

[alestark]

Main confusion comes from:

or the worst-case scenario with both PPPoE and VLAN involved, you would theoretically need an MTU of 1512 on the ethernet port, 1508 on the VLAN created on it and then 1500 for PPPoE. This is the third case in the appended image.

which makes me thinking that both enp1s0 and vmbr1 shall be set on 1512. This would reflect on vtnet0 inside the vm with the same value.
Then I can config via /etc/rc.conf.local (didn't found any other way from the UI) the MTU for the vtnet0_vlan835 to 1508 and finally to configure 1500 in pppoe0. So far, clear.
But then:

1. The OpnSense settings here are somewhat "wrong". If you have a WAN over PPPoE over VLAN, you "should" have to set WAN MTU = 1500, pppoe0 = 1508, ethernet port = 1512

which, to me, is not clear. "WAN MTU" is intended to be the vlan? becase if it's vtnet0 it's in contrast to what said before...
Lastly

PPPOEVLAN MTU: 1508 (if needed in your setup).

again, before, was suggested to set to 1500.

[meyergru]

What is not clear? You should have to set pppoe0 to 1500, but this results in a smaller real MTU size. So, to actually generate 1500 on pppoe0, you must set 1508. That is the exact point what I meant when I told you about the difference of the value you set vs. the values you get for PPPoE interfaces. In this case, the distinction is vital.

With a PPPoE connection, your (logical) WAN interface is obviously pppoe0, not any VLAN. Maybe your ISP asks the underlying interface to be on a VLAN or maybe he does not.

It is just as easy as depicted in the article: You want to get MTU 1500 on WAN, which is pppoe0. In order to actually get that, you need to set 1508, just because it does not work as it should with PPPoE.

Then, in order to have the underlying VLAN (if your ISP needs it) accomodate a net 1500 MTU over PPPoE, you have to add 8 PPPoE header bytes to the net MTU of 1500 bytes, thus the VLAN needs 1508 bytes. In this case, you can also set that, because that works as expected, so set value = get value here.

The underlying ethernet interface beneath the VLAN has to accomodate the 4 byte VLAN header on top of that, so you need 1512 bytes in total. So you set 1512 bytes on the NIC, which, in your case, is vtnet0. Obviously, if there is no VLAN, you can skip the 4 bytes and only set 1508 on the underlying physical interface.

And because you have vtnet0 on a bridge, which connects to a physical NIC on your Proxmox host, you will have to set 1512 (or 1508, depending on VLAN or not) bytes on both of those, as well.

[alestark]

Thanks for your patience.
I have set values according to your suggestions:

Code Select Expand

root@OPNsense:~ # ifconfig vtnet0 | grep mtu
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
root@OPNsense:~ #

Code Select Expand

root@OPNsense:~ # ifconfig vtnet0_vlan835 | grep mtu
vtnet0_vlan835: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
root@OPNsense:~ #

Code Select Expand

root@OPNsense:~ # ifconfig pppoe0 | grep mtu
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
root@OPNsense:~ #
Proxmox side:

Code Select Expand

[root@pve-02]: ~ # ifconfig enp1s0 | grep mtu
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1512
[root@pve-02]: ~ #

Code Select Expand

[root@pve-02]: ~ # ifconfig vmbr2 | grep mtu
vmbr2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1512
[root@pve-02]: ~ #
Then i tried with the pings from within the firewall:

Code Select Expand

root@OPNsense:~ # ping -4 -c4 -D -s 1472 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1472 data bytes
1480 bytes from 8.8.8.8: icmp_seq=0 ttl=117 time=5.410 ms
1480 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=5.976 ms
1480 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=4.952 ms
1480 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=5.074 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.952/5.353/5.976/0.397 ms
root@OPNsense:~ #

Code Select Expand

root@OPNsense:~ # ping -4 -c4 -s 1480 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1480 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # My understanding is that the above ping should result differently according to the first post?

Code Select Expand

root@OPNsense:~ # ping -4 -c4 -D -s 1480 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1480 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ #

[meyergru]

No, everything seems right. 1472 bytes payload work (this is 1500 bytes MTU for ping) and 1480 bytes do not, as expected. Even 1474 bytes should return an error when the packet is transmitted without fragmentation.

You can allow fragmentation for the test ping, but I got mixed results as to if that actually fragments the packet or just drops it - newer kernels seem to drop them. You can experiment with MSS clamping for that, but in fact I never had any problems when "ping -D -s 1474" works. I updated the article to reflect that.

[alestark]

Thanks a lot man :) appreciate your patience.