PPPoE MTU issues - no ICMP message too big being sent

[duh_au]

Hi all,

I have 19.1.3 installed on a VM with a PPPoE connection to my ISP.
If I leave the MTU setting default (blank) is calculates the PPPoE interface MTU to 1492 - that is correct.
If I ping from the LAN side through this interface with DF (Don't Fragment) set a frame at 1492 passes ok, as expected, but if you go past this size, I would expect an ICMP message generated by the firewall indicating that the packet is too big for the interface (should be 1492) .. but I am never seeing this message.

Bug?

Dave

[His.Dudeness]

Hi Dave

I’ve also got an PPPoE dial-up on my FTTH and have noticed exactly the same behavior. As well with packets going through IPsec or OpenVPN site to site tunnels (which also adds overhead to the packets)

If I send a packet out to the internet with DF bit set and it is to big for the MTU at my outside interface I would expect an ICMP Code 4 (Destination Unreachable Fragmentation Needed, DF Set ) Instead the packet just times out. 

This is how I tested:
If I ping a host from my windows machine within my LAN, the biggest packet I can send without fragmentation is 1500.
That would be “ping 192.168.1.1 -f -l 1472” 
28 byte for the ICMP echo request and 1472 byte payload = 1500 byte.

If I add one more byte (1473) I get an answer (I guess from my NIC driver) that the packet cannot go through because it is to big. If I was using a real application it would have know to decrease packet size until the packet fits through (path mtu discovery)

My outside interface of the OPNSense has an MTU of only 1492 to begin with, because it does PPPoE. 
If I send a ping out to the internet with a payload size between 1464 and 1472 I should get a response from OPNsense that the packet won’t fit though the WAN interface without fragmentation.  But my Ping just times out. I think if there is no ICMP Code 4 reply, path mtu discovery cannot work which leads to issues with (at least) UDP-based services.

I got a floating firewall rule allowing any ICMP from any to any on all interfaces so ICMP should not get dropped by the firewall.


Michael

[Zenspartan]

This error is happening on our opnsense boxes since the upgrade too, we are using intel NICs. It’s like something is failing with the mss clamping, we are having issues with TCP too ( websites aren’t loading) MTU is hovering around the 1410 mark

When enabling/disabling interfaces of intrusion detection it works for 5 seconds though, weird

[duh_au]

I tried to get this fixed by lodging a bug - https://github.com/opnsense/core/issues/3328, but clearly the person who picked it up had no clue and I don't have time to waste going in circles.
While it all looks nice with a new frontend compared to pFSense .. at least pFSense just worked as expected.

[franco]

If you can't fix the bug yourself please do not make the assumption that others who neither have your setup nor the immediate ability to fix your bug are "a waste of time". Or at least consider that this may go both ways.

We've always said it and say it again: use what works. That's why you have options.


Cheers,
Franco

[His.Dudeness]

Hi

sorry for asking again, but can anyone confirm if this is a bug? Or rather a configuration error in my setup?

To the question whether this is only an issue with Ipsec: IMHO the missing ICMP code 4 response is messing with IPsec more then with other traffic.

In my case the transfer rate via IPsec site-to-site drops to 300KB/s. I guess due to high packet loss.

When I set the MTU on the NICs of client and server on both ends of the site-to-site manually to 1300 (via netsh) transfer speed increases to 6 MB/s

When I try the same with an UDP based OpenVPN  tunnel (without adjusting the MTU of client and server) the transfer rate is much higher (4-5 MB/s)  BUT the OpenVPN process is using almost all of my CPU.
Even on a CPU with AES-NI and even if I disable encryption completely in OpenVPN

My suspicion would be, that OpenVPN just handles fragmentation and reassembling transparently. Traffic through the tunnel is therefore less affected by packet loss but consumes lots of CPU

I can even see that my “ping 192.168.1.1 -f -l 1472”  goes through the OpenVPN tunnel, although it could never "fit" due to the tunnel overhead and PPPoE. So OpenVPN seems to just ignore the DF bit, fragments and reassembles the packets. (which is better than what the ipsec tunnel does but also far from ideal)

cheers
Michael