PPPOE MSS claimping problem

[Deltorek112]

So I followed you instructions and got 1476 using the mentioned script
And set the MTU to this values
pppoe0      1476
vlan0.35    1484
igc0        1486
But is is still not loading some https sites

Can I use "opnsense-revert -kr 25.1.12" to revert to least know working version from 25.7.1?

[meyergru]

You are clearly miscalculating some values and/or misreading the instructions.

If you have PPPoE on top of a VLAN, you will have some MTU X on your igc0 (physical interface), then, in theory, you will have 4 bytes less on the VLAN and 8 bytes less for PPPoE, so the difference between the MTU on igc0 and pppoe0 cannot be 10, but should be 12.

Also, the script takes all of this into account and tells you the final (i.e. PPPoE) MTU.

So, if it says you can do 1476 at most, you should set:

pppoe0 1476
vlan0.35 1484
igc0 1488

This is very strange, however, because almost any physical interface can do 1500 (which would not hurt in your situation, either).

I suspect that your initial measurement was done after you already mucked around with some of these values.

The instructions tell you to even enlarge the physical MTU by 12 bytes (if you interface supports it) in order to keep 1500 on the PPPoE interface, because that causes less problems in the long run.

Also, keep in mind that you cannot reliably set all of these values dynamically. They sometimes work only after a proper reboot.

[Deltorek112]

So I set MTU to:
pppoe0      1500
vlan0.35   1508
igc0      1512

Restarted router (pc with opnsense)

Run the script and got MTU of 1492
And set MTU to this:
pppoe0      1492

Code Select Expand

pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
        description: WAN (opt6)
        options=0
        inet 100.73.222.158 --> 213.158.195.232 netmask 0xffffffff
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>vlan0.35   1500

Code Select Expand

vlan0.35: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: WAN_VLAN (opt8)
        options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG>
        ether 02:76:c6:01:35:0b
        groups: vlan
        vlan: 35 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc0      1504

Code Select Expand

igc0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1504
        description: WAN_INTERFACE (opt7)
        options=4e427bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 02:76:c6:01:35:0b
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Another restart of router + windows PC

And with MTU on PC being set to 9000 I still have problems with some https sites
They go away if it is set to 1400

Does any one know if the command "opnsense-revert -kr 25.1.12" can be used to revert to working version?
Or do I have to reinstall and restore?

[Monviech (Cedrik)]

But PMTU should work correctly.

Here is my PPPoE with VLAN, no MSS or MTU settings has been adjusted anywhere. It's just standard ICMP:

Code Select Expand

16:35:35.811489 IP opn-dev-01.gast.example.org.35152 > lb-140-82-121-4-fra.github.com.ssh: Flags [.], seq 22:1482, ack 772, win 64948, length 1460
16:35:35.811568 IP 172.16.1.1 > opn-dev-01.gast.example.org: ICMP lb-140-82-121-4-fra.github.com unreachable - need to frag (mtu 1492), length 576
16:35:35.811571 IP opn-dev-01.gast.example.org.35152 > lb-140-82-121-4-fra.github.com.ssh: Flags [P.], seq 1482:1590, ack 772, win 64948, length 108
16:35:35.811612 IP 172.16.1.1 > opn-dev-01.gast.example.org: ICMP lb-140-82-121-4-fra.github.com unreachable - need to frag (mtu 1492), length 576
16:35:35.811961 IP opn-dev-01.gast.example.org.35152 > lb-140-82-121-4-fra.github.com.ssh: Flags [.], seq 22:1474, ack 772, win 65535, length 1452
It tries to communicate with MSS 1460 first, gets a need to frag (mtu 1492), and then tries again with MSS 1452

MSS 1460 fails
MSS 1452 works

Client caches the 1452 value and uses it for all connections.

[meyergru]

You obviously did not follow the instructions in the linked article closely:

As an example, if you have a WAN over PPPoE over VLAN, you should have to set WAN MTU = 1500, PPPOE VLAN = 1508, ethernet port = 1512, and it really only works for me with these tightly controlled MTUs:

pppoe0 MTU: 1500 (this must only be set in the advanced settings of pppoe0, not on the WAN interface itself, see screendumps). This will set the WAN MTU.
ONT MTU: (this means the physical ethernet port): 1512 if you have a VLAN for PPPoE, 1508 if not.
PPPOE VLAN MTU: 1508 (if needed in your setup).

To be able to set these values, you will have to assign each of the underlying interfaces a name, which you would normally not need to.

Also, set the above values in the web UI and then reboot, if in doubt - the values sometimes cannot be successfully changed via UI manipulations, because the order of application seems to be wrong that way.

Afterwards, verify that the shown MTUs are as expected via "ifconfig". Also verify that the achievable MTU over your WAN connection works with the tool I linked above.

The result you are seeing seems to indicate you only set some of the values or in the wrong place.

I see:

Code Select Expand

igc3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
...
igc3_vlan40: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
...
pppoe0: flags=10089d1<UP,POINTOPOINT,RUNNING,NOARP,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500

And I have set igc3 to 1512, pppoe0 to 1500 (in the advanced settings of the PPPoE interface, NOT on WAN!), and you cannot set the VLAN MTU manually.

P.S.: PMTU works correctly under the assumptions Monviech made, i.e. on the OpnSense side. Whether the destination supports it, is a whole different story, which is why I recommend to simply enlarge the resulting MTU to 1500 bytes (that works more often than not).

[mbk2]

I've now compared two Wireshark tcpdumps. One was captured while I was still experiencing connection issues. The destination in both cases is a FritzBox connected via IPsec. I believe the behavior is similar with other destinations as well (for testing, I accessed the German adac.de website).

In the problematic dump, the connection starts with an MSS of 1460, which is then reduced to 1374. However, with this MSS, no data is transmitted.

In comparison, the tcpdump I captured today also starts with an MSS of 1460 but gets reduced to 1366. In this case, the connection works correctly, and the website loads as expected.

It seems that something is off with automatic MSS clamping. When neither MTU nor MSS is explicitly set, the connection fails. However, once I manually configure MTU 1500 and MSS 1452 on the WAN interface (not the pppoe interface in advanced), the connection works reliably.

[Deltorek112]

I have set it up as in screenshots right now

[meyergru]

...so completely different from what I told. There is literally no screendump that shows the correct settings.

[Deltorek112]

Yes, first I used values you have provided, it did not work
then I executed the script to calculate MTU and it gave me 1492
I used it to calculate the MTU of other ones
PPPoE0 MTU 1492
VLAN0.35 MTU 1500
IGC0 MTU 1504

Is this wrong with the script output?
If yes could you tell me what values should I use? And where to set them?

[meyergru]

Again: The script output will show you what is effective after you have set whatever you like. 1492 will almost always be the default if you set nothing at all. Theoretically, it should be 1488 with both PPPoE and VLAN, but in fact, the 4 bytes VLAN are neglegible, so 1492 still works.

So, in order to enlarge the WAN MTU effectively to 1500 bytes, you have to set the values as instructed. Iff that fails and after a reboot you still only get 1492, you may be one of the rare cases where your ISP does not support enlarging the ethernet MTU on the raw interface to 1512 bytes. This rarely happens, but YMMV.

I use the settings as given and it looks like this:

You cannot view this attachment.
You cannot view this attachment.
You cannot view this attachment.

I do not set the VLAN MTU explicitely.

You can see that my ifconfigs who the expected sizes and the script output is 1500 for me (because my ISP allows it).

[Deltorek112]

I set it up exactly like in your screenshots, restarted the router, and I still get the broken behavior of forever loading page for at least this sites:
poczta.wp.pl
mail.google.com

[meyergru]

What does the script say after the settings and what does ifconfig show?

[Deltorek112]

Script output:
./mtuScript.sh poczta.wp.pl
./mtuScript.sh: line 1: $: command not found
Maximum MTU size: 1474

./mtuScript.sh mail.google.com
./mtuScript.sh: line 1: $: command not found
Maximum MTU size: 1228

Ifconfig:

Code Select Expand

igc0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
        description: WAN_ONT (opt4)
        options=4e427bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 02:76:c6:01:35:0b
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Code Select Expand

vlan0.35: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG>
        ether 02:76:c6:01:35:0b
        groups: vlan
        vlan: 35 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Code Select Expand

pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: WAN (opt6)
        options=0
        inet 100.74.2.8 --> 213.158.195.232 netmask 0xffffffff
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

[meyergru]

Well, yes, the call syntax of the find_mtu script is for a known IPv4 that supports PMTU, like 8.8.8.8, so poczta.pl does not even work here, because it is reached via IPv6. Also, the line1: error indicates that you did not download the correct script or it was converted with incorrect line-endings or something.

Same goes for mail.google.com, it is also resolved to an IPv6. Use IPv4s, not names, like "./mtuScript.sh 8.8.8.8", but verify that the script content is valid beforehand.

The ifconfig output looks about right.

[Deltorek112]

For IPv6 I don't have it, my ISP does not support it, and I blocked it in config.
As for script for 8.8.8.8:
./mtuScript.sh 8.8.8.8
Maximum MTU size: 1474