[SOLVED] Asynchronous IPv6 routing problem

[kreilinger]

Hi,

I have the following setup:

• Vultr VM with IPv4 + IPv6 connectivity announcing my IPv6 prefix via BGP and a Wireguard tunnel to my on-premise
• OpnSense, where I have IPv6-only VLAN interface "TEST" which contains a single
• debian Test VM, with a static IPv6 address.

What works:

• debian VM has Internet connectivity, can ping everywhere, do downloads via http, etc., all good
• debian VM can do ping6 and traceroute6 to any destination

What does not work:

• Inbound traffic to the debian VM (traffic initiated from wan, tested with ping6, traceroute6 and HTTP

I did a tcpdump on the Vultr VM and saw the traffic gets sent into the tunnel.
I did a tcpdump on the OpnSense and saw the traffic arriving from the tunnel and leaving the VLAN interface.
I did a tcpdump on the debian VM and saw the traffic arriving from the OpnSense.
I did a tcpdump on the OpnSense and saw the answer arriving on the VLAN interface.

Now the problem: The answer packages do not appear on the tcpdump for the Wireguard tunnel on the OpnSense, but instead i see them on my pppoe0 wan interface.

Why is this happening and how do I achive that answer traffic gets sent back to the interface it originated from?

[wbk]

Hi Kreilinger,

Your subject states 'routing problem', did you have a look at the configured routes and their metrics? I have no idea in this case, but that is where I'd start my investigation.

Secondly, out of interest, which goal are you trying to reach with this setup?

[kreilinger]

I don't have IPv6 connectivity so I am announcing my own subnet via BGP and the Vultr VM and use its connection so my local networks can have IPv6 internet.

I know I can set different gateway priorities, but shouldn't the firewall know that this is answer/followup traffic and use the gateway it originated from? It works perfectly if the traffic originates from my local network. Whats the difference when it originates from outside?

[wbk]

> I don't have IPv6 connectivity so...
Great motivation! :-)

> ... shouldn't the firewall know that this is answer/followup traffic and use the gateway it originated from?
I think that is not how it works, that is, matching incoming and outgoing gateways, but maybe with an actual understanding of the topic could chime in.

>  It works perfectly if the traffic originates from my local network.
With 'it' being what? You mean "IPv6 routing going out via the (only) gateway that has IPv6"? Not trying to be obtuse here, I'm trying to get a picture of things, and with my limited network knowledge I can not fill in the blanks you leave in the story :-P

Edit - would your scenario not be similar to a multi-WAN-setup when you define your Vultr-link as WAN? Or WAN-failover for IPv6? In another topic, https://forum.opnsense.org/index.php?topic=33032.msg159805#msg159805 , that seems to be resolved with some help.

[kreilinger]

If the traffic originates from my local network, the traffic matches the correct firewall rule that has the IPv6 tunnel gateway set and followup traffic gets correctly routed back to its original source.

My wan pppoe connection does not even have an IPv6 link local address. Still if e.g. i ping one of my hosts from an external site, the icmp echo request arrives at my local VM, but the icmp echo reply gets sent out via OpnSense's pppoe0 interface (which is v4 only).

> would your scenario not be similar to a multi-WAN-setup when you define your Vultr-link as WAN? Or WAN-failover for IPv6?

I don't know. If one link is v4 only and the other is v6 only, do I need to configure a multi wan setup? My testnetwork is IPv6 only.

[opnsense-user123]

...your issue could be similar to mine, https://forum.opnsense.org/index.php?topic=33057.0

Please let us know if you figure it out!

[wbk]

If the traffic originates from my local network, the traffic matches the correct firewall rule that has the IPv6 tunnel gateway set and followup traffic gets correctly routed back to its original source.

Can you see that happening when you follow the firewall log?

I have the luxery of both native (for lack of a better word; my ISP furnishes me with both) IPv4 and IPv6 on my connection, but I lack fluency in BSD. My routes for IPv4 and IPv6 seem, respectively:

Code Select Expand


# netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            lo0-3.bras1.fi001. UGS      pppoe0
osba.nl            link#8             UHS         lo0
localhost          link#3             UH          lo0
172.26.0.0/16      link#1             U           em0
vpoort             link#1             UHS         lo0
dns1.freedom.nl    lo0-3.bras1.fi001. UGHS     pppoe0
lo0-3.bras1.fi001. link#8             UH       pppoe0
dns2.freedom.nl    lo0-3.bras1.fi001. UGHS     pppoe0

Internet6:
Destination        Gateway            Flags     Netif Expire
default            fe80::6a22:8eff:fe UGS      pppoe0
localhost          link#3             UHS         lo0
dns1.freedom.nl    fe80::6a22:8eff:fe UGHS     pppoe0
dns2.freedom.nl    fe80::6a22:8eff:fe UGHS     pppoe0
2d49-3781-2a10.con localhost          UGSB        lo0
2a10-3781-2d49.con link#1             U           em0
vpoort             link#1             UHS         lo0
fe80::%em0/64      link#1             U           em0
fe80::6c34:c2ff:fe link#1             UHS         lo0
fe80::%lo0/64      link#3             U           lo0
fe80::1%lo0        link#3             UHS         lo0
fe80::%pppoe0/64   link#8             U        pppoe0
fe80::4003:eb6:a62 link#8             UHS         lo0
fe80::6c34:c2ff:fe link#8             UHS         lo0


In the above:

• em0 is my LAN
• pppoe0 is my IPv4 connection through vlan6 on em1
• IPv6 probably uses IPv4 connectivity (through pppoe0, but things break when I tick the box 'use IPv4 connectivity' in the IPv6 configuration

I'd expected some metric listed here, as does ip route in Linux, but these things work just a bit different. The metric on pppoe0 is 0:

Code Select Expand

# ifconfig  pppoe0
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
        description: WANpoort (wan)
        inet6 fe80::4003:eb6:a62:d503%pppoe0 prefixlen 64 scopeid 0x8
        inet6 fe80::6c34:c2ff:feb8:147c%pppoe0 prefixlen 64 scopeid 0x8
        inet 45.138.52.95 --> 185.93.175.233 netmask 0xffffffff
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>


For Linux the most specific route with the lowest metric has priority. I'd imagine the same goes for FreeBSD.

I have no guess why outgoing traffic that originates in your LAN can find its way out of the IPv6 tunnel, while replies to traffic outside of your LAN try to get out of your regular connection.

[zan]

I usually fix asymmetric routing by forcing the reply-to.
Try setting the "reply-to" field in your IPv6 pass rule on your WG interface to the WG gateway.

[kreilinger]

zan, you are a genius!

On the inbound rule on my WG interface I set reply-to to my WG gateway and everything works now!

Thank you so much!

[zan]

You are welcome.
Also I just re-read your posts, since your WG gateway is the only IPv6 upstream in your system you can just set up a static default IPv6 route (::/0) via your WG gateway.