26.1.1 MTU Issues on PPPoE

[Taunt9930]

Hi all, I upgraded from 25.7.11_9 directly to 26.1.1 as I elected to wait for the *.1 patch before I made the move.

Immediately upon upgrading, my internet has become 'troublesome', generally laggy with slow notifications from my cameras etc. One thing that is entirely repeatable is speedtests are significantly reduced, when they run. If I try speedtest.net it seems to take ages finding the optimal server, and when I run the speedtest I get about 400-600 down on a 950 connection, after it taking a while to start. On occasion, it fails to run the speedtest at all and throws an error.

If I go back to my 25.7.11_9 snapshot, all is well again. If I return to the 26.1.1 snapshot, back to the same problems consistently.

I have had a poke about, and cannot see anything obvious. But, consistently 25.7.11_4 is fine, and 26.1.1 has these issues. I run a pppoe connection with Zen in the UK. MTU for the WAN Interface is set to 1508 (for Calculated PPP MTU:1500) as it has always been. Changing this to 1500 made no difference - for some reason it smelt a bit like an MTU issue. I have disabled any shapers/pipes, Zenarmor and no different. Going back to 25.7.9_4 solves it instantly.

Where can I start looking / what can I do to try and narrow down the issue? I am keen to work it out and stay on 26.1.1. Thanks.

Upload appears not to be affected

[Taunt9930]

I am pretty sure this is an IPv6 MSS/MTU issue I'm seeing on 26.1, that I did not see/was not there in previous versions. I noticed speedtest.net was resolving to an IPv6 address and did some more investigation.

I have changed nothing, but 26.1.1 breaks something. I have 1508 set for my WAN PPPoE MTU (Calculated 1500 PPP) - does 26.x do something different on the IPv6 MTU/MSS with this that it wasn't doing before or not doing something now that it was in previous versions?

Do I have to manually clamp the IPv6 MTU/MSS now somehow/somewhere on 26.1 that wasn't required previously? I did nothing on 25.7 and had no issues.

on 25.7:

• ping -6 -l 1452 google.com - Pings get returned.

However on 26.1:

• ping -6 -l 1452 google.com - I get a timeout (packet silently dropped)
• ping -6 -l 1440 google.com - Pings get returned.

Any help would be appreciated - is this a behaviour change I now need to mitgate with some additonal settings, or a bug?

Thanks.

EDIT: Could it actually be IPv4 also? Thinking about it, I seemed to have issues on VLANs/interfaces that do not have IPv6 enabled. I have gone back to 25.7.9_4 for now so can't check.

I remember some time ago there was some shenanigans with setting an MTU of 1508 and it not being correctly applied to the interface /you had to do something else at the parent interface level, until the implementation was changed to fix it.

[Taunt9930]

Changed the title - my issues seem to be across IPv4 and IPv6 and relate to the MTU implementation (somehow) on 26.1.1. I have 1508 MTU Set on my PPPoE WAN Interface (Calculated PPP: 1500) and that has worked forever. When I upgrade to 26.1.1 I have MTU/MSS related issues. I can flip back and forth from 25.7.11_9 to 26.1.1 and the issue is entirely repeatable on 26.1.1.

If I run the Path MTU Discovery Test, it illustrates the issue - http://pmtud.enslaves.us/

Attached screenshots

25.7.11_9 - IPv4 MSS OK to 1460, and IPv6 MSS OK to 1440 (as expected)

26.1.1 - IPv4 MSS 536, and IPv6 MSS 1220

Something is amiss and this is entirely repeatable.

Has something changed in the implememtation of Mini Jumbo on the WAN Interface?

[pfry]

Have default sysctls/tuneables changed? Check (shell: "sysctl net.inet.tcp" or "sysctl [individual setting]"):

net.inet.tcp.mssdflt: 1460
net.inet.tcp.v6mssdflt: 1440
net.inet.tcp.minmss: 536
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.pmtud_blackhole_detection: 0
net.inet.tcp.pmtud_blackhole_mss: 1460
net.inet.tcp.v6pmtud_blackhole_mss: 1440

Those are... not my current values, but what I expect to set when I change firewalls. Not sure if I'll enable blackhole detection - need to read up on it. (Configuring the blackhole MSS settings explicitly is generally not necessary.) Of course these normally only affect TCP terminating on the firewall - I would not expect them to affect sessions initiated from clients behind the firewall. Clients would be affected by an MSS clamp but those generally use explicit values (in the interface settings) and are not derived from system MSS settings. Set them on OPNsense under "System: Settings: Tunables". (The v6 settings may be configured from the v4 settings by default, but I haven't checked that.)

I don't think these are your actual issue, but it can't hurt to check 'em.

[meyergru]

I always configure this like shown here, which also uses full 1500 bytes MTU.

My result looks like your first picture, i.e.:

Direction Tested	Maximum Size	Segment	Client Sent	MSS	Notes
Server to Client	IPv4	1460	1460	OK
Client to Server	IPv4	unlimited	(n/a)	OK
Server to Client	IPv6	1440	1440	OK
lient to Server	IPv6	unlimited	(n/a)	OK

Code Select Expand

got in probe for mss 536 (max seg 1460)
got in probe for mss 1460 (max seg 1460)
got in probe for mss 1460 (max seg 1460)
finished in probing, maximum mss 1460 peer mss 1460 initial peer mss 1460
got out probe for mss 520
got out probe for mss 1461
got out probe for mss 9000
finished out probing, maximum mss 9000
got in probe for mss 536 (max seg 1440)
got in probe for mss 1440 (max seg 1440)
got in probe for mss 1440 (max seg 1440)
finished in probing, maximum mss 1440 peer mss 1440 initial peer mss 1440
got out probe for mss 520
got out probe for mss 1441
got out probe for mss 9000

And there are my exact settings (note that after changing those values, the results will only be reproducable after a reboot, because the order of settings matter (i.e. they influence one another)).

You cannot view this attachment.

You cannot view this attachment.

You cannot view this attachment.

You cannot view this attachment.

[abulafia]

same config (dual stack, IPv6 via PPPoE) gives me lower MTUs:

Code Select Expand

Direction Tested Maximum Size Segment Client Sent MSS Notes
Server to Client IPv4 1452 1452 OK
Client to Server IPv4 unlimited (n/a) OK
Server to Client IPv6 1432 1432 OK
Client to Server IPv6 unlimited (n/a) OK


Code Select Expand

got in probe for mss 536 (max seg 1452)
got in probe for mss 1452 (max seg 1452)
got in probe for mss 1452 (max seg 1452)
finished in probing, maximum mss 1452 peer mss 1452 initial peer mss 1452
got out probe for mss 343
got out probe for mss 1453
got out probe for mss 9000
finished out probing, maximum mss 9000
got in probe for mss 536 (max seg 1432)
got in probe for mss 1432 (max seg 1432)
got in probe for mss 1432 (max seg 1432)
finished in probing, maximum mss 1432 peer mss 1432 initial peer mss 1432
got out probe for mss 343
got out probe for mss 1433
got out probe for mss 9000
finished out probing, maximum mss 9000
PINGs seem to transmit the "full" MTU 1500, though (both from a client and from OPNsense):

Code Select Expand

ping -4 -c4 -D -s 1472 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 1472(1500) bytes of data.
[1770722028.061190] 1480 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=5.63 ms
[1770722029.062243] 1480 bytes from 1.1.1.1: icmp_seq=2 ttl=58 time=4.75 ms
^C

[Taunt9930]

Thanks Meyergru - maybe I'll try replicating your settings, minus the VLAN as that is not required.

That being said, setting the WAN MTU on the first page to 1508 directly was enough up until now (after it changed where you didn't need to separately set up the parent interface) - it seems something has changed in 26.1 that means the more detailed setting might be required....

[meyergru]

same config (dual stack, IPv6 via PPPoE) gives me lower MTUs:

1. Same config as who?
2. Note the caveat to reboot after having applied the settings.
3. When you read the HOWTO closely, you will find that this does not work in all cases - especially, your ISP must support it. The safe value to set MTU to with PPPoE (regardless of VLAN) is 1492.

[Taunt9930]

I am at a loss. I have just got home, and further looked at your setup meyergru. I realised that this is how I originally had it set up (as  was required before) - Create/Assign the igb0 (in my case) physical interface and explicitly set a 1508 MTU (I have no VLAN requirement), then on the PPPoE point to point link that sits on it configure the link MTU as 1500.

There was an update to OPNSense (23.7.5 ish?), where it was stated that explicitly creating the parent interface was no longer required and setting the MTU to 1508 on the [WAN] interface would correctly set the adaptor/physical interface MTU and the Child PPPoE link (as per the calculated MTU shown @1500) - this has worked since then.

So, I have recreated the config as per yours (and rebooted) - with the parent/physical interface and I still have the same issues. I get the sub-optimal results on the Path MTU Discovery Test, and the internet is 'broken' - sites loading slowly, 30-50% line rate on the download, speedtest.net failing to run the test at all. Upload not affected.

Small bit of digging:

• ifconfig shows (physical interface) igb0: at 1508 MTU
• ifconfig shows pppoe0: at 1500 MTU
• ping -D -s 1472 google.com returns 1480 bytes
• ping6 -D -s 1452 google.com returns 1460 bytes

so I can see the MTU is set correctly . But still, I have those results and everything feels 'broken'.

Roll back to 25.7.11_9 and all is well, but I see no difference in the config or what ifconfig reports. 

Is there anything I can do to try and narrow down the cause? I don't want to be stuck on 25.7 for all time.....

I'm hoping someone might see this, and recognise *something* that changed in 26.1 that may affect this, even if it is a really weird edge case specific to my use-case. I can then (hopefully) mitigate it - but as far as I can see, there is nothing different/special about my WAN setup.

[Boxer]

Are you using DNSmasq Router Advertisements or RADVD? I was seeing some peculiarities on some websites unless I explicity set the RA MTU in DNSmasq. Switch to RADVD (if you're using DNSmasq RA) and see if your issues persist

[Taunt9930]

Are you using DNSmasq Router Advertisements or RADVD? I was seeing some peculiarities on some websites unless I explicity set the RA MTU in DNSmasq. Switch to RADVD (if you're using DNSmasq RA) and see if your issues persist

I'm using RADVD. I am sure I tried explicitly setting the MTU there also. Did something change between 25.7.11_9 and 26.1 in the Router Advertisements - was the issue you saw only after 26.1 upgrade?

Issues are across IPv4 and IPv6 - they persist if I disable IPv6 on my client machine.

[Boxer]

I don't know for sure because I only switched to DNSmasq RA after 26.1. I've since switched back to radvd which fixed all my problems

[Boxer]

I'm on BT PPPOE (Openreach) and just set the wan mtu to 1508 without any issues on websites. Are you on Zen Openreach or Zen City Fibre?

[Taunt9930]

I'm on BT PPPOE (Openreach) and just set the wan mtu to 1508 without any issues on websites. Are you on Zen Openreach or Zen City Fibre?

Openreach. As said, been running that config with no issues since around 23.7.5 and as per Meyergru (with physical interface explicitly stated) before that.

Upgrade to 26.1 breaks something. I've done it twice now, same result.

[meyergru]

This does not look like an MTU issue if you can use those ping sizes - it look just fine.

Did you also use traffic shaping? Maybe the old ISP had lower speeds and you shape it to fit? Happened before...

Just saw that you disabled all shaping...

No idea what could be wrong.