My system:
*fully updated OPNsense 24.10.2-amd64 running on a Protecli VP4670.
*WAN configuration: pppoe->vlan->igc1 (connects to an ONT via ethernet)
*LAN configuration: vlan->igc0
*I do not manually set MTU parameters on any interface.
*I do use shaping on WAN and LAN
OPNsense correctly calculates the WAN MTU as 1492 (=1500-8 = 1464+28). Indeed, executing on the OPNsense box:
(FP1) ping -D -s 1464 9.9.9.9 # receives a ping response
(FP2) ping -D -s 1465 9.9.9.9 # is rejected by the interface with a "ping: sendto: Message too long" message. No message is actually sent down the wire.
On a linux machine connected to the LAN interface, executing:
(LP1) ping -M do -D -s 1464 9.9.9.9 # receives a ping response
(LP2) ping -M do -D -s 1465 9.9.9.9 # receives no response
From within OPNsense a packet capture of both the LAN interface and the WAN interface shows that the
echo request from ping LP2 is seen in the LAN interface capture but is NOT present in the WAN interface capture.
But the Firewall log for the WAN interface shows that the LP2 ping message is "passed" to the WAN interface for transmission.
I deduce that OPNsense (or FreeBSD) is discarding the LP2 ping without generating the ICMP type=3 code=4 response that should be sent to the device on the LAN.
Is there some setting (tunable?) that I can use to persuade OPNsense to send the ICMP type=3 code=4 responses to my LAN devices?
Maybe you should first take a look at this (https://forum.opnsense.org/index.php?topic=45658.0). Otherwise, you would have to set your LAN MTU to the same value as your WAN or use MSS clamping, but needless to say, enlarging the WAN MTU to effective 1500 bytes is way easier.
Thanks for the fast response.
I had read those articles but my question regards what seems to me to be a bug in FreeBSD/OPNsense (or more likely a user error of course).
Surely if the MTU of the WAN is 1492 and the LAN is 1500 then OPNSense should generate a ICMP type=3 code=4 response and send it to the LAN device.
My evidence shows that this is not happening.
Or is my understanding incorrect?
Yes, it sure seems so. If your LAN MTU is 1500 and you send a packet that is loo long (i.e. ping -s 1465), it cannot be transmitted on the WAN interface, so you will never receive an answer. Matter-of-fact, as is explained in my HOWTO, if you use both VLAN and PPPoE encapsulation, your effective WAN MTU will be 1500-8-4 = 1488. Since a ping adds 28 bytes, you normally can only transfer 1460 bytes.
The fact that you can send 1464 bytes may be due to the fact that there seem to be some miscalculations around VLANs and PPPoE interface MTU lengths in OPNsense. For example, when you install a VLAN on a parent interface, OPNsense will still think the VLAN MTU is the same as the parent interface MTU (which might work, but does not have to). Also, sometimes, the shown pppoe MTU is 1508, sometimes it is 1500. See the thread for examples of different results. All of that is irrelevant. What matters is the maximum packet size you can send over your WAN interface. Add 28 to that and that is your effective MTU. Thus, if you can "ping -s 1472 8.8.8.8", everything is fine.
BTW: Your connected Linux machine will gladly send 1500 byte packets and they will be received on OpnSense LAN. Alas, they cannot be forwarded. But that is nothing that the ping command on the Linux host would notice. Consider the case when the packet gets lost anywhere else on the routing path - fact of the matter is that you will not get a response, which has nothing to do with if you can send a request.
All I can say is repeat:
1. Try if you can get effective MTU of 1500 on WAN by using the instructions given, then you do not have to worry about a different LAN MTU.
2. If that is not possible - for whatever reason (and there are many) - you only can:
a. Reduce the LAN MTU to the same value as your WAN MTU or
b. Use MSS clamping to get what you expect. However, note that the packets will be split into one larger packet and one very short one, which leads to much reduced speeds in some cases.
Quote from: sja1440 on March 08, 2025, 07:13:09 PMSurely if the MTU of the WAN is 1492 and the LAN is 1500 then OPNSense should generate a ICMP type=3 code=4 response and send it to the LAN device.
It does.
$ ping -D -s 1465 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1465 data bytes
576 bytes from opnsense (192.168.1.1): frag needed and DF set (MTU 1492)
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 d505 f290 0 0000 3f 01 715e 192.168.1.129 8.8.8.8
Request timeout for icmp_seq 0
576 bytes from opnsense (192.168.1.1): frag needed and DF set (MTU 1492)
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 d505 6db8 0 0000 3f 01 f636 192.168.1.129 8.8.8.8
Packet trace:
22:30:52.708813 IP 192.168.1.129 > dns.google: ICMP echo request, id 23689, seq 3, length 1473
22:30:52.710851 IP opnsense > 192.168.1.129: ICMP dns.google unreachable - need to frag (mtu 1492), length 576
22:30:53.714018 IP 192.168.1.129 > dns.google: ICMP echo request, id 23689, seq 4, length 1473
22:30:53.716262 IP opnsense > 192.168.1.129: ICMP dns.google unreachable - need to frag (mtu 1492), length 576
^C
But only if the DF bit is set which ping does not do by default. But TCP does if the initiating host supports PMTUD.
Yep, it does, but from Linux you have to use "ping -M do -s 1465" - or in my case:
#ping -M do -s 1473 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 1473(1501) bytes of data.
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
ping: local error: message too long, mtu=1500
But I use 25.1.2, not 24.10. And I am not quite sure what would happen if WAN MTU < LAN MTU. I think the problem might silently be ignored, like with any further hop that dismisses these packets.
Quote from: meyergru on March 08, 2025, 10:50:18 PMBut I use 25.1.2, not 24.10. And I am not quite sure what would happen if WAN MTU < LAN MTU. I think the problem might silently be ignored, like with any further hop that dismisses these packets.
Same same - 25.1.2. Pinging from my Mac. LAN MTU on my OPNsense is 1500, WAN MTU is 1492. So OPNsense is responding perfectly correct with "frag needed but DF set".
Then the only question that remains why the OP does not get that response on his equipment, which seems much the same MTU configuration as yours. The only difference seems to be 24.10. vs. 25.1.2.
Interesting that OpnSense detects the difference on the first hop - I would not have expected that. For good measure and to fight braindead PMTUD hops along the way, I always use both WAN and LAN MTU to be 1500.
Quote from: Patrick M. Hausen on March 08, 2025, 10:33:12 PMQuote from: sja1440 on March 08, 2025, 07:13:09 PMSurely if the MTU of the WAN is 1492 and the LAN is 1500 then OPNSense should generate a ICMP type=3 code=4 response and send it to the LAN device.
It does.
Thanks to you both for your comments and to Patrick for confirming that OPNsense 25.1.2 does indeed respect the requirements of RFC 1191.
In the meantime I have carried out further tests after having:
* disabled Suricata
* disabled shaping
* inserted explicit fw rules to accept (and log) icmp destination unreachable messages (Should be pointless because such messages are associated with the original ping and so should automatically be accepted without logging)
In each case I did the following from my linux machine:
ping -c 1 -M do -D -s 1464 151.101.0.81 # Returns a echo response and so verify that the target host does honour pings
ping -c 1 -M do -D -s 1465 151.101.0.81 # Should receive icmp type 3 code 4 response from OPNsense, but doesnt.
In no case did the LAN and WAN interface captures show any icmp type 3 code 4 responses. The evidence seems to point to OPNsense 24.10 silently dropping the "oversize" DF marked icmp after being passed by the firewall on WAN but before being sent down the wire.
I have looked through the change logs of OPNsense releases since 24.10 and I see that 25.1.1 includes the following change:
"
src: pf: send ICMP destination unreachable fragmentation needed when appropriate"
So it looks as if the non-compliance with RFC 1911 is fixed in 25.1.1 - which would explain why neither of you have the problem.
Thanks again for your help.