Failed: IPv6 fragmented packet delivery

[giovino]

Greetings,

In the United States using Xfinity Internet I see failed IP fragmented packet delivery over IPv6 using OPNsense 17.7.5.

You can run a test here:

http://icmpcheckv6.popcount.org/

Reference:
https://blog.cloudflare.com/ip-fragmentation-is-broken/

Could someone else using OPNsense 17.7.x with a IPv6 connection run the test at http://icmpcheckv6.popcount.org/ and report your results? Specifically I see:

IP fragmented packet delivery
✗ The request timed out. Looks like IP fragments failed to be delivered to you.

If I use curl + tcpdump I see:

Code: [Select]

curl -v -s http://icmpcheckv6.popcount.org/frag -o /dev/null
*   Trying 2a01:7e01::f03c:91ff:fe16:a2e9...
* TCP_NODELAY set
* Connected to icmpcheckv6.popcount.org (2a01:7e01::f03c:91ff:fe16:a2e9) port 80 (#0)
> GET /frag HTTP/1.1
> Host: icmpcheckv6.popcount.org
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sat, 14 Oct 2017 05:49:38 GMT
< Content-Type: text/plain; charset=utf-8
< Connection: close
< Transfer-Encoding: chunked
<
{ [14 bytes data]
* Recv failure: Connection reset by peer
* stopped the pause stream!
* Closing connection 0


Code: [Select]

tcpdump -ni igb0 '(ip[6] & (1<<5)) != 0 or (ip[7] != 0) or (ip[6] & ((1<<5)-1) != 0) or ip6[6] == 44'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:49:38.585841 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 87111905:87112385, ack 4046851107, win 224, options [nop,nop,TS val 1674343770 ecr 2705732609], length 480: HTTP
01:49:38.616794 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 1428:1908, ack 1, win 224, options [nop,nop,TS val 1674343770 ecr 2705732609], length 480: HTTP
01:49:38.647635 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 2856:3336, ack 1, win 224, options [nop,nop,TS val 1674343770 ecr 2705732609], length 480: HTTP
01:49:38.678546 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 4284:4764, ack 1, win 224, options [nop,nop,TS val 1674343770 ecr 2705732609], length 480: HTTP
01:49:38.709258 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 5712:6192, ack 1, win 224, options [nop,nop,TS val 1674343770 ecr 2705732609], length 480: HTTP
01:49:38.739918 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [P.], seq 7140:7620, ack 1, win 224, options [nop,nop,TS val 1674343770 ecr 2705732609], length 480: HTTP
01:49:39.004806 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [P.], seq 7140:7620, ack 1, win 224, options [nop,nop,TS val 1674343896 ecr 2705732747], length 480: HTTP
01:49:39.405000 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 0:480, ack 1, win 224, options [nop,nop,TS val 1674344016 ecr 2705732747], length 480: HTTP
01:49:40.205523 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 0:480, ack 1, win 224, options [nop,nop,TS val 1674344256 ecr 2705732747], length 480: HTTP
01:49:41.805125 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 0:480, ack 1, win 224, options [nop,nop,TS val 1674344736 ecr 2705732747], length 480: HTTP
01:49:45.111728 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 0:480, ack 1, win 224, options [nop,nop,TS val 1674345728 ecr 2705732747], length 480: HTTP
01:49:51.511640 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 0:480, ack 1, win 224, options [nop,nop,TS val 1674347648 ecr 2705732747], length 480: HTTP
01:50:04.311778 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 0:480, ack 1, win 224, options [nop,nop,TS val 1674351488 ecr 2705732747], length 480: HTTP
01:50:30.338806 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 0:480, ack 1, win 224, options [nop,nop,TS val 1674359296 ecr 2705732747], length 480: HTTP
01:51:21.539176 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > xxx: frag (0|512) 80 > 47493: Flags [.], seq 0:480, ack 1, win 224, options [nop,nop,TS val 1674374656 ecr 2705732747], length 480: HTTP

Thank you!

[phoenix]

Seems to be OK for me:

The web site test at:  http://icmpcheckv6.popcount.org/ gavve me a green box:

✓ All good! ICMP path MTU message was successfully delivered to you.

The curl test gave this:

Code: [Select]

curl -v -s http://icmpcheck.popcount.org/frag -o /dev/null
* Hostname was NOT found in DNS cache
*   Trying 139.162.188.91...
* Connected to icmpcheck.popcount.org (139.162.188.91) port 80 (#0)
> GET /frag HTTP/1.1
> User-Agent: curl/7.37.0
> Host: icmpcheck.popcount.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sat, 14 Oct 2017 06:33:23 GMT
< Content-Type: text/plain; charset=utf-8
< Connection: close
< Transfer-Encoding: chunked
<
{ [data not shown]
* Closing connection 0
...and this:

Code: [Select]

curl  -v -s http://icmpcheck.popcount.org/icmp --data @payload.bin
* About to connect() to icmpcheck.popcount.org port 80 (#0)
*   Trying 139.162.188.91...
* Connected to icmpcheck.popcount.org (139.162.188.91) port 80 (#0)
> POST /icmp HTTP/1.1
> User-Agent: curl/7.29.0
> Host: icmpcheck.popcount.org
> Accept: */*
> Content-Length: 8100
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Date: Sat, 14 Oct 2017 06:39:44 GMT
< Content-Type: text/plain; charset=utf-8
< Connection: close
< Transfer-Encoding: chunked
<
{"msg1": "Upload complete", "mtu":1500, "lost_segs":0, "retrans_segs":0, "total_retrans_segs":0, "reord_segs":3, "snd_mss":1448, "rcv_mss":853}
* Closing connection 0

.... and tcpdump gave this:

Code: [Select]

tcpdump -ni eth0  '(ip[6] & (1<<5)) != 0 or (ip[7] != 0) or (ip[6] & ((1<<5)-1) != 0) or ip6[6] == 44'   
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:29:26.318081 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (0|512) 80 > 35338: Flags [.], seq 998043207:998043687, ack 1367675351, win 232, options [nop,nop,TS val 1675060106 ecr 23061848], length 480: HTTP
08:29:26.318109 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (512|512)
08:29:26.318112 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (1024|221)
08:29:26.348688 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (0|512) 80 > 35338: Flags [.], seq 1213:1693, ack 1, win 232, options [nop,nop,TS val 1675060106 ecr 23061848], length 480: HTTP
08:29:26.348723 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (512|512)
08:29:26.348727 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (1024|221)
08:29:26.379326 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (0|512) 80 > 35338: Flags [.], seq 2426:2906, ack 1, win 232, options [nop,nop,TS val 1675060106 ecr 23061848], length 480: HTTP
08:29:26.379360 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (512|512)
08:29:26.379363 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (1024|221)
08:29:26.409898 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (0|512) 80 > 35338: Flags [.], seq 3639:4119, ack 1, win 232, options [nop,nop,TS val 1675060106 ecr 23061848], length 480: HTTP
08:29:26.409934 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (512|512)
08:29:26.409937 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (1024|221)
08:29:26.440536 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (0|512) 80 > 35338: Flags [.], seq 4852:5332, ack 1, win 232, options [nop,nop,TS val 1675060106 ecr 23061848], length 480: HTTP
08:29:26.440572 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (512|512)
08:29:26.440575 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (1024|221)
08:29:26.471076 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (0|512) 80 > 35338: Flags [.], seq 6065:6545, ack 1, win 232, options [nop,nop,TS val 1675060106 ecr 23061848], length 480: HTTP
08:29:26.471111 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (512|512)
08:29:26.471114 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (1024|221)
08:29:26.501395 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (0|512) 80 > 35338: Flags [P.], seq 7278:7758, ack 1, win 232, options [nop,nop,TS val 1675060106 ecr 23061848], length 480: HTTP
08:29:26.501411 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (512|512)
08:29:26.501414 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (1024|80)
With my limited knowledge, that all seems correct to me.

[giovino]

Phoenix,

First and foremost, thank you for helping me out.

I have a few questions:

for the website check I too get Green for the first check:

ICMP path MTU packet delivery
✓ All good! ICMP path MTU message was successfully delivered to you.

It's the second check I see fail:

IP fragmented packet delivery
✗ The request timed out. Looks like IP fragments failed to be delivered to you.

Second, their webpage is a not 100% correct, the curl command you ran was IPv4

curl -v -s http://icmpcheck.popcount.org/frag -o /dev/null
* Hostname was NOT found in DNS cache
*   Trying 139.162.188.91...

Can you run curl and tcpdump using the ipv6 hostname?

curl -v -s http://icmpcheckv6.popcount.org/frag -o /dev/null

Note: I see your tcpdump results seem to show IPv6 so I am a touch confused

...
08:29:26.318081 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (0|512) 80 > 35338: Flags [.], seq 998043207:998043687, ack 1367675351, win 232, options [nop,nop,TS val 1675060106 ecr 23061848], length 480: HTTP
08:29:26.318109 IP6 2a01:7e01::f03c:91ff:fe16:a2e9 > 2a01:e0a:6e:c111:be5f:f4ff:fe62:99c5: frag (512|512)
...

Thank you!

[phoenix]

Sorry I missed the send output from the blackhole check, the second also worked:

✓ All good! IP fragments were successfully delivered to your host.

My mistake with the IPv4 (I am a novice at this , here's the IPv6 output:

curl -v -s http://icmpcheckv6.popcount.org/frag -o /dev/null
* Hostname was NOT found in DNS cache
*   Trying 2a01:7e01::f03c:91ff:fe16:a2e9...
* Connected to icmpcheckv6.popcount.org (2a01:7e01::f03c:91ff:fe16:a2e9) port 80 (#0)
> GET /frag HTTP/1.1
> User-Agent: curl/7.37.0
> Host: icmpcheckv6.popcount.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sat, 14 Oct 2017 17:38:33 GMT
< Content-Type: text/plain; charset=utf-8
< Connection: close
< Transfer-Encoding: chunked
<
{ [data not shown]
* Closing connection 0

Is that everything? If I missed something or there's anything else you need just let me know. BTW, I'm based in France and my ISP is free.fr.

[giovino]

Phoenix,

Thank you, that is what I was looking for.

This seems to be local to my configuration as I had other people in the United States on the same ISP but non OPNsense users report their results and it worked for them too.

So... something in my Opnsense configuration OR something local to the region for my ISP which seems unlikely. If anyone has any ideas, I'd love to hear them.

Thanks!

[phoenix]

I don't know if this will be of any use but I have seen mentioned elsewhere that the scrub option might need to be disabled, have you tried that (it's under: Disable interface scrub option under Firewall/Settings/Normalization) I also don't know if the NIC offload settings would have any effect on this. FWIW, my OPNsense install is a VM and I have the scrub setting still enabled and all the offload functions are disabled.