Before filing a bug report, I rather ask here: Under System: Firmware, you can start a connectivity audit. For me this gives:
***GOT REQUEST TO AUDIT CONNECTIVITY***
Currently running OPNsense 25.1.3 (amd64) at Wed Mar 12 13:13:59 CET 2025
Checking connectivity for host: pkg.opnsense.org -> 89.149.222.99
PING 89.149.222.99 (89.149.222.99): 1500 data bytes
--- 89.149.222.99 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Checking connectivity for repository (IPv4): https://pkg.opnsense.org/FreeBSD:14:amd64/25.1
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 862 packages processed.
Updating mimugmail repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: ........ done
Processing entries: .......... done
mimugmail repository update completed. 193 packages processed.
All repositories are up to date.
Checking connectivity for host: pkg.opnsense.org -> 2001:1af8:5300:a010:1::1
PING(1548=40+8+1500 bytes) 2001:a61:47f:b942::42 --> 2001:1af8:5300:a010:1::1
--- 2001:1af8:5300:a010:1::1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
Checking connectivity for repository (IPv6): https://pkg.opnsense.org/FreeBSD:14:amd64/25.1
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 862 packages processed.
Updating mimugmail repository catalogue...
pkg: https://opn-repo.routerperformance.net/repo/FreeBSD:14:amd64/meta.txz: Unknown resolver error
repository mimugmail has no meta file, using default settings
pkg: https://opn-repo.routerperformance.net/repo/FreeBSD:14:amd64/packagesite.pkg: Unknown resolver error
pkg: https://opn-repo.routerperformance.net/repo/FreeBSD:14:amd64/packagesite.txz: Unknown resolver error
Unable to update repository mimugmail
Error updating repositories!
Checking server certificate for host: opn-repo.routerperformance.net
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = E5
verify return:1
depth=0 CN = opn-repo.routerperformance.net
verify return:1
DONE
Checking server certificate for host: pkg.opnsense.org
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = RapidSSL TLS RSA CA G1
verify return:1
depth=0 CN = pkg.opnsense.org
verify return:1
DONE
***DONE***
I see a problem with this:
The ping being called for both IPv4 and IPv6 targets is almost impossible to succeed, because it is being called via "ping -c4 -s1500". Even with a normal ethernet connection without VPN, VLAN or PPPoE overhead, the maximum payload size that is feasible is 1472 for IPv4 and 1452 bytes, respectively, so these pings will almost never work. So, shouldn't the "-s" option be removed altogether?
The "unknown resolver" problem is just because the mimugmail repository does not have an IPv6 address, so we can disregard that.
Since this came up a number of times allow me to say: "oh no, not this topic again ;("
The short answer: no. The long answer is somewhere in GitHub and forums.
Ah, I found it.
You try to force ping fragmentation by specifically using a too large payload. Alas, that does not work for my one of my ISPs, although I tried MSS clamping by setting the MSS value of the WAN interface.
That fragmentation works for DG with DHCP, but not for M-Net with PPPoE/VLAN.
I know that these tests are overbearing, but will highlight issues when firmware updates don't work at all.
The idea is: cannot update please help -> provide audit output -> give appropriate help.
Cheers,
Franco
No problem with that, Franco.
What still bothers me is that I see that IP fragmentation does not work for pppoe devices, while it does for normal ethernet (and VLAN interfaces). It does not seem to be an ISP problem, but something in OpnSense/FreeBSD going wrong.
When I create a ping request with an oversized payload and do a tcpdump on any outgoing interface, I can see split packets for igcX, for example, but not for pppoe0 (there is only one outbound packet capped at the MTU size). I played around with TCPMSSfix and MSS clamping to no avail.
If that worked as expected with PPPoE, I would not have seen any errors in the first place...
Quote from: meyergru on March 12, 2025, 03:25:37 PMI played around with TCPMSSfix and MSS clamping to no avail.
What is that supposed to achieve for anything but TCP? :-)
Yes, you are right, Patrick... What I wonder is if that non-fragmenting behavior is:
a. a configuration problem on my part
b. a bug in FreeBSD/OpnSense (IDK why it should not be possible to fragment IP packets on PPPoE interfaces)
c. a feature, because it should not be possible on PPPoE interfaces for reasons I do not see
Quote from: meyergru on March 12, 2025, 02:21:41 PM[...]
You try to force ping fragmentation by specifically using a too large payload. Alas, that does not work for my one of my ISPs, although I tried MSS clamping by setting the MSS value of the WAN interface.
That fragmentation works for DG with DHCP, but not for M-Net with PPPoE/VLAN.
As an enhancement, why not 500 or 1200B payloads with a "checking connectivity" label, then, say, 5000B with "checking fragmentation"? Or would that just confuse folks?
Quote from: meyergru on March 12, 2025, 04:15:26 PMYes, you are right, Patrick... What I wonder is if that non-fragmenting behavior is:
a. a configuration problem on my part
b. a bug in FreeBSD/OpnSense (IDK why it should not be possible to fragment IP packets on PPPoE interfaces)
c. a feature, because it should not be possible on PPPoE interfaces for reasons I do not see
Works as designed for me. OPNsense 25.1.3. Telekom PPPoE, MTU 1492.
I send a UDP frame with a payload of 1472 bytes - that is the maximum that fits in one 1500 byte frame (20 bytes IP and 8 bytes UDP header):
sudo nping --udp -p 5555 my.server.somewhere --data-length 1472I see these leave my Mac in single frames:
16:57:19.857832 IP (tos 0x0, ttl 64, id 51683, offset 0, flags [none], proto UDP (17), length 1500)
192.168.1.129.53 > *.*.*.*.5555: 1679 update NXDomain*- [3640q], [|domain]
The source port seems to be npings choice to better penetrate firewalls. That's definitely my packet here.
Then at the same time on OPNsense, interface pppoe0:
16:57:19.903223 IP (tos 0x0, ttl 63, id 51683, offset 0, flags [+], proto UDP (17), length 1492)
*.*.*.*.15311 > 66.135.24.72.5555: UDP, length 1472
16:57:19.903236 IP (tos 0x0, ttl 63, id 51683, offset 1472, flags [none], proto UDP (17), length 28)
*.*.*.* > *.*.*.*: ip-proto-17That's an initial fragment of length 1472 (this time including the UDP header to account for 20 bytes IP header and the PPPoE MTU of 1492 - the numbers just match by accident) followed by another fragment of 28 bytes in length to transmit the full 1500 bytes IP+UDP+garbage frame.
So OPNsense when forwarding a UDP packet from 1500 bytes MTU Ethernet to 1492 bytes MTU PPPoE does correctly fragment the frames.
Let's first not mix up different tools/protocols and passing/originating through/from OpnSense and/or IPv4 vs. IPv6. When I do as you say (and it looks as if you are use nping on a client in your LAN), I get completely different results, maybe because of specific settings. Depending on which endpoints I use, I can even see "Message too long".
What do you get when you try "ping -4 -c 4 -s 1500 pkg.opnsense.org" from the OpnSense CLI?
That should very specifically call for an oversized packet (whatever your exact MTU size is) and be fragmented.
When I do just that for a DG setup with DHCP only, I see:
# ping -4 -c 4 -s 1500 pkg.opnsense.org
PING pkg.opnsense.org (89.149.222.99): 1500 data bytes
1508 bytes from 89.149.222.99: icmp_seq=0 ttl=56 time=9.722 ms
1508 bytes from 89.149.222.99: icmp_seq=1 ttl=56 time=11.407 ms
1508 bytes from 89.149.222.99: icmp_seq=2 ttl=56 time=12.613 ms
1508 bytes from 89.149.222.99: icmp_seq=3 ttl=56 time=9.247 ms
--- pkg.opnsense.org ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 9.247/10.748/12.613/1.343 ms
As I wrote, I can see long and short packets out the WAN interface.
When I do it for M-Net with PPPoE over VLAN, I see:
# ping -4 -c 4 -s 1500 pkg.opnsense.org
PING pkg.opnsense.org (89.149.222.99): 1500 data bytes
--- pkg.opnsense.org ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
In that case, tcpdump shows only the larger part of the packets, capped at my specific MTU size going out the WAN interface.
The moment I reduce the payload to 1472 bytes (with my effective MTU being 1500 bytes, such that no fragmentation is needed), I get correct answers.
Quote from: meyergru on March 12, 2025, 05:25:10 PMWhat do you get when you try "ping -4 -c 4 -s 1500 pkg.opnsense.org (https://pkg.opnsense.org/)" from the OpnSense CLI?
root@opnsense:~ # ping -4 -c 4 -s 1500 pkg.opnsense.org
PING pkg.opnsense.org (89.149.222.99): 1500 data bytes
1508 bytes from 89.149.222.99: icmp_seq=0 ttl=56 time=28.136 ms
1508 bytes from 89.149.222.99: icmp_seq=1 ttl=56 time=28.178 ms
1508 bytes from 89.149.222.99: icmp_seq=2 ttl=56 time=28.276 ms
1508 bytes from 89.149.222.99: icmp_seq=3 ttl=56 time=28.102 ms
--- pkg.opnsense.org ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 28.102/28.173/28.276/0.065 ms
And correctly fragmented (otherwise it could hardly work):
root@opnsense:~ # tcpdump -n -i pppoe0 host 89.149.222.99
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), snapshot length 262144 bytes
17:27:57.930488 IP *.*.*.* > 89.149.222.99: ICMP echo request, id 43692, seq 0, length 1472
17:27:57.930505 IP *.*.*.* > 89.149.222.99: ip-proto-1
17:27:57.958217 IP 89.149.222.99 > *.*.*.*: ICMP echo reply, id 43692, seq 0, length 1472
17:27:57.958466 IP 89.149.222.99 > *.*.*.*: ip-proto-1
17:27:57.958473 IP 89.149.222.99 > *.*.*.*: ip-proto-1
My modem is doing the VLAN tagging. And if I am not mistaken MTU should be 1500 even with the VLAN (and line MTU 1508 accordingly) dropping to 1492 for PPPoE. Possibly there's a bug if OPNsense does the 802.1q encapsulation?
Yes, I think that there must be something in that area. Actually, what I do is documented here (https://forum.opnsense.org/index.php?topic=45658.0).
There was always something off with that approach, and now it is getting very weird... What I was doing is:
Physical interface (igc3), MTU: 1512
VLAN interface (igc3_vlan40), MTU: 1508
PPPoE interface (pppoe0), MTU: 1500
This is to ensure that the effective MTU on the WAN interface stays at 1500. To do that, you have to add 4 bytes for VLAN and 8 for PPPoE headers.
BUT: To set these MTUs, you have to tweak a bit. For example, you cannot set the VLAN MTU directly. I only got this to work by setting 1508 MTU on the WAN (pppoe0) interface itself. It also shows that value, but testing with "ping -D" shows that indeed the effective MTU is 1500.
Now for the fun part: tcpdump shows this for "-s 1472" (which maxes out the packet size):
# tcpdump -i pppoe0 icmp and host pkg.opnsense.org
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), snapshot length 262144 bytes
18:04:44.860421 IP xxx > 89.149.222.99: ICMP echo request, id 27230, seq 0, length 1480
18:04:44.875720 IP 89.149.222.99 > xxx: ICMP echo reply, id 27230, seq 0, length 1480
18:04:45.870270 IP xxx > 89.149.222.99: ICMP echo request, id 27230, seq 1, length 1480
18:04:45.885280 IP 89.149.222.99 > xxx: ICMP echo reply, id 27230, seq 1, length 1480
18:04:46.876792 IP xxx > 89.149.222.99: ICMP echo request, id 27230, seq 2, length 1480
18:04:46.891115 IP 89.149.222.99 > xxx: ICMP echo reply, id 27230, seq 2, length 1480
18:04:47.887232 IP xxxe > 89.149.222.99: ICMP echo request, id 27230, seq 3, length 1480
18:04:47.902393 IP 89.149.222.99 > xxx: ICMP echo reply, id 27230, seq 3, length 1480
^C
8 packets captured
But with "-s 1474" (which is 2 bytes too much), it shows:
18:06:56.488712 IP xxx > 89.149.222.99: ICMP echo request, id 5938, seq 0, length 1482
And then, no answers. So, there is no fragmentation, as I said.
And now with "-s 1480" (i.e. 8 bytes more than 1500 bytes MTU), still:
18:06:58.488712 IP xxx > 89.149.222.99: ICMP echo request, id 5938, seq 0, length 1488
But with "-s 1482" (i.e. 10 bytes more than 1500 bytes MTU), it finally stays at that size - this is true also for even larger payloads:
18:06:59.488712 IP xxx > 89.149.222.99: ICMP echo request, id 5938, seq 0, length 1488
So, it seems that although the effective MTU on PPPoE is apparently 1500 bytes, the shown MTU of 1508 is the one that effectively limits the outbound packet size.
Plus, even when that limit is reached, the packets are still not being fragmented here - and I am at a loss why that is.
I will experiment with an unmodified setup with 1492 bytes MTU and see how that goes, but probably, you are right, Patrick and 802.1q coming into play breaks things. Alas, I cannot easily change where tagging is done, because I have an ONT and no DSL modem.
And there you have it: The moment I change the pppoe0 MTU manually to 1500 with an otherwise unmodified setup, I get answers regardless of the packet length...
So I will have to find a way to stack up the MTUs of all involved interfaces such that the pppoe0 MTU will show 1500 (but not limit the effectively usable MTU to 1492).
To conclude this from my part:
To change all physical, VLAN and PPPoE interface MTUs, you must create named interfaces to be able to fully control both MTU and MSS (which is really MTU). I have updated my instructions accordingly (https://forum.opnsense.org/index.php?topic=45658.0). Only after doing that, fragmentation works correctly. I have the impression that somehow probably the order of interface creation matters, because in the linked thread, some people reported correct WAN MTU without tightly controlling all layer MTUs - yet for me, that did not work.
Fragmentation works that way with IPv4, but not with IPv6. As expected, the maximum ping packet size is 1452 bytes (i.e. 20 bytes lower because of higher IPv6 overhead), but no fragmentation occurs.
On a side note: When I set none of these MTU or MSS values and stay at the default 1492 MTU for pppoe0, everything works - only at lower MTU limits. Both IPv4 and IPv6 do fragmentation.
I give up. Experimenting with that is tedious, because you cannot change the values and just test the results, because they often change after a reboot.
I think @pfry's comment is nice, if the tests were done with different payloads, one could see if ping in general and fragmentation works.
Quote from: meyergru on March 13, 2025, 09:20:07 AMBoth IPv4 and IPv6 do fragmentation.
I assume when sending packets from OPNsense, right? Because for IPv6 intermediate systems must not fragment but send an ICMPv6 "packet to big" instead. Only the source is permitted to split the packets in fragments if the application in question cannot simply use a smaller payload.
Kind regards,
Patrick
Yes, I always try with ping -s from OpnSense itself.
Correcting myself after a few reboots: This is not at all reproducable. It still seems to be dependend on random initialisation of interfaces and I cannot detect any pattern in it. Even without any specific MTU settings on any layer, I sometimes do not get fragmentation for IPv6.
I filed a bug, because now I now that the order of interface activation matters....
https://github.com/opnsense/core/issues/8435