Cannot ping IPv6 gateway but traceroute works

[andreaslink]

I have a strange IPv6 behavior running my OPNsense 20.7.2-amd64 with IPv6 behind a FritzBox.

I got an IPv6 address as well as /60 sub net assigned to my WAN, but when I try to ping the gateway directly from the firewall, all ICMPv6s get lost. I had opened firewall for all ICMPv6 on WAN on all directions.

This is what happens:

fe80::c225:6ff:feff:820d = FritzBox Link local address, correctly set as default IPv6 gateway
bce0 = WAN infterface

I cannot directly ping my router aka FritzBox :

root@OPNsense:~ # ping6 -c 3 fe80::c225:6ff:feff:820d%bce0
PING6(56=40+8+8 bytes) fe80::221:5eff:fec8:be88%bce0 --> fe80::c225:6ff:feff:820d%bce0

--- fe80::c225:6ff:feff:820d%bce0 ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss


I cannot ping google's ipv6 dedicated host:

root@OPNsense:~ # ping6 -c3 ipv6.google.com
PING6(56=40+8+8 bytes) 2a02:2f4:xxxx:xxxx:221:5eff:fec8:be88 --> 2a00:1450:4001:81b::200e
--- ipv6.l.google.com ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss


BUT I can make a fine traceroute6 to that address, that works as expected (done via UDP):

root@OPNsense:~ # traceroute6 ipv6.google.com
traceroute6 to ipv6.l.google.com (2a00:1450:4001:81b::200e) from 2a02:2f4:xxxx:xxxx:221:5eff:fec8:be88, 64 hops max, 20 byte packets
 1  2a02:2f4:xxxx:xxxx:c225:6ff:feff:820d  0.510 ms  0.468 ms  0.388 ms
 2  2a02:2f0:0:72::  4.589 ms  14.459 ms  21.283 ms
 3  2a02:2f0:0:34::  4.682 ms  4.618 ms  4.451 ms
 4  2a02:2f0:4002::5d32:a0  7.877 ms  4.728 ms  4.649 ms
 5  2001:4860:0:12e6::4  5.377 ms
    2001:4860:0:12e3::3  4.899 ms
    2001:4860:0:12e4::2  5.358 ms
 6  2001:4860::c:4001:ec6  5.069 ms
    2001:4860::c:4001:ebe  15.328 ms
    2001:4860::c:4001:ec6  4.939 ms
 7  2001:4860::c:4001:9920  15.494 ms
    2001:4860::c:4001:5c4  10.797 ms
    2001:4860::c:4001:9920  15.498 ms
 8  2001:4860::8:0:cb95  14.999 ms
    2001:4860::c:4000:f873  14.720 ms *
 9  2001:4860::1:0:d0d8  15.346 ms
    2001:4860::9:4001:31f1  14.559 ms  14.683 ms
10  2001:4860:0:1::673  14.393 ms  14.732 ms
    2001:4860:0:1::671  14.432 ms
11  fra15s16-in-x0e.1e100.net  14.496 ms
    2001:4860:0:1::671  14.465 ms  14.405 ms


Then I just ask for all the router in my local network via multicast request and I suddenly get an answer from the Fritzbox , this really puzzles me:

#All Routers Address:
root@OPNsense:~ # ping6 -c 2 ff02::2
PING6(56=40+8+8 bytes) fe80::221:5eff:fec8:be88%bce0 --> ff02::2
16 bytes from fe80::c225:6ff:feff:820d%bce0, icmp_seq=0 hlim=64 time=0.562 ms
16 bytes from fe80::c225:6ff:feff:820d%bce0, icmp_seq=1 hlim=64 time=0.629 ms

--- ff02::2 ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.562/0.595/0.629/0.034 ms


So it obviously can receive and answer on ICMPv6, so I ping it directly again, but it does not answer:


root@OPNsense:~ # ping6 -c 3  fe80::c225:6ff:feff:820d%bce0
PING6(56=40+8+8 bytes) fe80::221:5eff:fec8:be88%bce0 --> fe80::c225:6ff:feff:820d%bce0

--- fe80::c225:6ff:feff:820d%bce0 ping6 statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss


I really need a good advise, what is wrong here?

Summary:

• Multicast ping aka ICMPv6 works
• Direct ping does not work
• traceroute6 into internet works fine
• ping into internet does not work

[banym]

Can you explain how you do your IPv6 routing?
Do you delegate a prefix to the OPNsense or how is your setup with the Fritzbox and your OPNsense?

Can you please add a network plan, too?

[andreaslink]

Thanks banym for your fast reaction, I hope my following words answer your questions.
 
I'm connected via FTTH behind a FritzBox (FB) and running IPv4 fine. I also got IPv6 some weeks ago within a dual stack setup and a common /56 to delegate and so to use on my own, which is assigned towards my FB. Within the FB IPv6 setup I activated "DNS-Server und IPv6-Präfix (IA_PD) zuweisen" aka allowing it to share (parts of) the /56 via DHCPv6 further on with other routers in the LAN. This works so far. Behind the FB I've my OPNsense (OPNsense 20.7.1-amd64) running, where all my clients are connected to.

It is set up with:

• Activated IPv6 witin OPNsense
• Set "IPv6 Configuration Type" on WAN (bce0) IF to DHCPv6
• Set within the basic "DHCPv6 client configuration":
• Request only an IPv6 prefix --> true
• Prefix delegation size --> 60 (As I got a /56 and I just wanted to have "some" (4 Bytes aka 16) subnets available on OPNsense (some more I can experiment on another router later)
• Send IPv6 prefix hint --> true
• Use IPv4 connectivity --> false
• Use VLAN priority --> Disabled
• On the LAN interface (bce1) I defined "IPv6 Configuration Type" as "None" (before I tested with "Track Interface" and further setup, turned it off for now, until IPv6 works towards WAN, before I start announcing the prefix into LAN)
• Deactivated "Block private networks" as well as "Block bogon networks" on LAN IF (as the LAN behind the FB obviously falls under these rules)
• Setup a Firewall rule to allow all ICMPv6 travel IN from WAN as well as for LAN (to cover all IPv6 ping and MTU-size requirements etc.)

With this setup, WAN got a decent IPv6 assigned from the FB as well as the /60. So this works fine and looks OK so far and as far as I can evaluate.

Please see the network plan, where I added the main parts:
 

            +------------------------------------------------------------------------------------+
            |                                                                                    |
            |                                    Internet                                        |
            |                                                                                    |
            +---------------------------------------+--------------------------------------------+
                                                    |
                                                    |
                                                    |
                                                    |
                      +-----------------------------+--------------------------+
                      |  FritzBox                                              |
                      |  fe80::c225:6ff:feff:820d                              |
                      |  Provider IPv6: 2a02:2f4:yyyy:yyyy:c225:6ff:feff:820d  |
                      |  IPv6-Prefix: 2a02:2f4:xxxx:xxxx::/56                  |
                      |  192.168.0.254                                         |
                      +---------------------+----------------------------------+
                                            |
                                            |
                                            |
                                            |
                      +---------------------+--------------------+
                      |  OPNsense                                |
                      |  WAN:                                    |
                      |   fe80::221:5eff:fec8:be88               |
                      |   2a02:2f4:xxxx:xxxx:221:5eff:fec8:be88  |
                      |   2a02:2f4:xxxx:zzzz::/60                |
                      |   192.168.0.100                          |
                      |  LAN:                                    |
                      |   DHCPv4: 192.168.123.10..100/24         |                     
                      |                                          |
                      +--+----+-------------------+---------+----+
                         |    |                   |         |
         +---------------+    |                   |         +-----+
         |                    |                   |               |
+--------+-------+    +-------+-------+   +-------+-------+   +---+---------+
| Client 1       |    | Client 2      |   | Client 3      |   |[...]        |
| 192.168.123.10 |    | 192.168.123.11|   | 192.168.123.12|   |             |
+----------------+    +---------------+   +---------------+   +-------------+


I also have further interfaces for IoT and Guests, but they are currently all setup comparable with LAN and no forther IPv6 subnets are delegated, so I skipped them in the drawing for now. And as said before, I first wanted to ensure that my WAN setup works with IPv6 and my OPNsense has full IPv6 connection before routing other IPv6 networks. Until then I stay with IPv4 for the LAN clients.

BTW: I have a wireguard VPN up and running on OPNsense as a working side2side connection, where the other side connects in via IPv6 directly onto the OPNsense wireguard service. This works without any problems as well.

Looking forward to further comments and hints .

[andreaslink]

And one more thing here to add. There is allegedly no route to the default gateway, what I cannot understand as the routing table clearly states, there is the route:

root@OPNsense:~ # traceroute6 fe80::c225:6ff:feff:820d (It does not matter, if I add "%bce0" or not.)
traceroute6 to fe80::c225:6ff:feff:820d (fe80::c225:6ff:feff:820d) from fe80::221:5eff:fec8:be88%bce0, 64 hops max, 20 byte packets
sendto: No route to host
 1 traceroute6: wrote fe80::c225:6ff:feff:820d 12 chars, ret=-1
 *sendto: No route to host
traceroute6: wrote fe80::c225:6ff:feff:820d 12 chars, ret=-1
 *sendto: No route to host
traceroute6: wrote fe80::c225:6ff:feff:820d 12 chars, ret=-1


But when I check the routing table, this is exactly how it should be and what I would expect

root@OPNsense:~ # netstat -nr
Routing tables

Internet:
[...]

Internet6:
Destination                       Gateway                       Flags     Netif Expire
default                           fe80::c225:6ff:feff:820d%bce0 UG         bce0
::1                               link#8                        UH          lo0
2a02:2f4:xxxx:xxxx::/64           link#1                        U          bce0
2a02:2f4:xxxx:xxxx:221:5eff:fec8:be88 link#1                    UHS         lo0
fd00:0:cafe:affe::/64             link#1                        U          bce0
fd00:0:cafe:affe:221:5eff:fec8:be88 link#1                      UHS         lo0
fe80::%bce0/64                    link#1                        U          bce0
fe80::221:5eff:fec8:be88%bce0     link#1                        UHS         lo0
fe80::%bce1/64                    link#2                        U          bce1
fe80::221:5eff:fec8:be8a%bce1     link#2                        UHS         lo0
fe80::%igb0/64                    link#3                        U          igb0
fe80::92e2:baff:fe68:cd74%igb0    link#3                        UHS         lo0
fe80::%igb1/64                    link#4                        U          igb1
fe80::92e2:baff:fe68:cd75%igb1    link#4                        UHS         lo0
fe80::%lo0/64                     link#8                        U           lo0
fe80::1%lo0                       link#8                        UHS         lo0

[banym]

Hast du die fe80 Addresse als Default-GW selbst gesetzt?
Nimm das mal bitte raus, die Route sollte doch über die Fritzbox über das Route-Advertisment kommen und nicht selbst gesetzt werden müssen.

[andreaslink]

Nein, natürlich nicht. Alles wurde automatisch vergeben, also wie es sich gehört und wie man es erwartet.

[andreaslink]

Ich habe noch eine Überlegung, könnte das ein Bug sein, dass das Interface fest bei der Link Local mit hinterlegt wird in der Routing Tabelle?

Warum "fe80::c225:6ff:feff:820d%bce0"? Das zu verwendende Interface steht ja am Ende der Routing Tabelle schon, warum auch in der Adresse? In der Routing Tabelle macht es doch auch als Link Local keinen Sinn.

Ich habe auch mal mit einem Linux Rechner hinter der FB vergleichen, da steht als default IPv6-Gateway auch die Link Local von der FB drin, allerdings ohne das Interface - sonst ist alles gleich.

Frage ob das wirklich ein Problem ist bei BSD? Und würde es dann auch zu dem "No route to host" error kommen?

[banym]

Oh we switched language ;-)
Should stay with English here.

In my case the local gateway looks similar to yours: fe80::e86:xx:xx%pppoe0
You can use netstat -nr or netstat -Sr to verify on the console.

Did you try to request a /64 network for testing on the opnsense. I don't see a big show stopper at the moment but don't have a fritzbox to verify the delegation mechanism here.

You allow ICMPv6 on the WAN interface?
You don't have "block private networks" and "bogon networks" enabled?

[andreaslink]

Hehe, agreed, you started with language switch, I'm flexible I just adapted .

But good to know, that this seems to be common in BSD with the interface being part of the link local IPv6 gateway IP and not being a bug.
A /64 I haven't tried as this would destroy the approach of separated sub nets. Just for testing I can give it a try.

And all your other questions are answered in my former post(s).
netstat output is visible in my example before, ICMPv6 firewall rule for WAN is in place (ping is also visible and green in FW log).
Bogon and RFC1918 is deactivated as WAN being in a common private network.

I also activated in "Firewall: Settings: Advanced" the option "Disable force gateway" as I read somewhere this might influence usage of routing table.

So the key question is left, what is needed to ensure OPNsense uses the route as announced in the routing table? Why does traceroute6 work, but ping6 cannot determine a route? What else is needed?

Would be nice to find someone with a compareable setup.


And second, this is something appearing from time to time in the log, could this influence this behaviour and is the root cause known? Curenly I guess this problem is independent, but I habe not done any more research yet.

error in configd communication Traceback (most recent call last): File "/usr/local/opnsense/service/configd_ctl.py", line 68, in exec_config_cmd line = sock.recv(65536).decode() socket.timeout: timed out

[andreaslink]

Any news here? Could anyone somehow prove that direct ping6 of gateway from OPNsense via IPv6 link local works?
Looking for someone running OPNsense behind a FritzBox with a delegated sub net prefix to compare - maybe in a working setup.

I simply don't get, why ICMPv6 does not find a route (as shown before), when routing table clearly states, it's the default route to go and traceroute also works as it should . I consider this as a bug or at least - if somewhere hidden - some in-transparent setup somewhere. I'm open to test anything to move on here.

Seems IPv6 support is not that sophisticated yet as IPv4 within OPNsense.

[banym]

The IPv6 stack is quite stable and works for me in different setups.
If I would have a similar setup I maybe could help more, but here I run it with VDSL from Telekom and that just works with direct PPOE and DHCPv6.

In your case I would debug by package capture to see if everything with ICMPv6 works a it should. Maybe for debugging turn off firewalling on the OPNsense for testing to see if then everything works as expected to reach the gateway.

[gratuxri]

Same problem here on OPNSense 21.10.3 Business Edition, that ipv6 gateway is unreachable, if firewall functionallity is on. Any suggestions. IPv6 ICMP is allowed for INPUT and OUTPUT on WAN interface.

WORKAROUND: System -> Gateways -> Single -> WAN_GWv6 disable, apply, enable, apply