Multi Wan, IPv6 & policy based routing problems / misunderstanding

[jbattermann]

Good evening,

I have some problems with setting up two static wan interfaces in combination with policy based routing and maybe someone sees / reads what's wrong and could push me in the right direction.

I have two physical WAN connections coming in, each with one static IPv4 (/30) and one static IPv6 address (/126). In addition to that one static IPv6 address per link, I got a /48 block assigned.. again.. per link and the two are not overlapping. One's WAN1, one's WAN2. WAN1's IPv4 and IPv6 gateways have highest priority and are the default route(s).

To my LAN interface I have assigned a static 192.168.x.x/24 IPv4 address and a static fd6e:XXXX:YYYY:ZZZZ::1/64 IPv6 address.


I do not have any plans for multi wan in the sense of failover or load balancing (at least for now), I simply want to route certain target networks over WAN2. I've therefore added two Firewall > Settings > LAN rules, one for the IPv4 and one for IPv6 target networks I want to route to using WAN2 to utilize the corresponding WAN2 gateways.

Now this seems to work, albeit something is.. wrong. I can e.g. traceroute to the target IPv4 networks and can see the WAN2 IPv4 GW is used, but for IPv6... all trace hops time out. And there are some other effects.. which I assume are related to IPv6 not working.


I can reproduce the issue i.e. when I take all Google ASNs and get their IPv4 and IPv6 ip ranges, put them in an alias group and setup a firewall rule the same way.. to route to these target networks over WAN2.. and everything works, but slow (google.com takes initially long to load, but then appears in an instant.. same with youtube and videos etc.) - which I think (and I obviously might be very wrong here), that whenever IPv6 doesn't work for google services and / or browsers, they fall "back" to IPv4, which seems to work just fine..


So long story short.. my question basically is - what pieces of the puzzle might I be missing here to perform IPv6 policy based routing over a non-default WAN interface?


Thanks,
-J

[5SpeedFun]

Under your Firewall -> Rules -> [Interface] are you choosing a gateway on the rule?

Also under Firewall -> Settings -> Advanced, did you "Disable reply-to" ?

[jbattermann]

Good evening!

Under your Firewall -> Rules -> [Interface] are you choosing a gateway on the rule?

Yes I do - I have two rules there, one for IPv4, one for IPv6 and each have the corresponding IPv4 and IPv6 GW for WAN2 set. 'reply-to' per rule is set to 'default' (I tried setting the GW there explicitly as well, but made no change)... and

Also under Firewall -> Settings -> Advanced, did you "Disable reply-to" ?

... nope, the 'Disable reply-to' checkbox is not checked. Should I?

[5SpeedFun]

Take a search through the forums for the implications of how this works.  This may fix your problem.

5SF

[jbattermann]

Take a search through the forums for the implications of how this works.  This may fix your problem.

5SF

Just to double check I look for the right thing.. you mean the '(Disable) reply-to' functionality, right?

[5SpeedFun]

Yes.

[bimbar]

Since you can not reasonably use NAT on IPv6, you must make sure that not only the firewall routes over the correct uplink, but also that the client devices use the right source address.

So I'm afraid the best you can do is policy route on the firewall so the right source address is routed over the right uplink, you will not be able to determine which uplink to use beyond that, since the client makes that decision.

[jbattermann]

Since you can not reasonably use NAT on IPv6, you must make sure that not only the firewall routes over the correct uplink, but also that the client devices use the right source address.

So I'm afraid the best you can do is policy route on the firewall so the right source address is routed over the right uplink, you will not be able to determine which uplink to use beyond that, since the client makes that decision.

Yes, I think we're thinking the same here.. the missing link / information for me is though how to tell OPNsense which of the two /64 IPv6 subnet belongs to which of the two WAN interface/gateway?

How and where can I configure that the one /64 is routed via WAN1 and the other /64 via WAN2? How does OPNsense choose (and let LAN clients know) that for a certain target IPv6 network, it has to go over WAN2 vs the default WAN1?

[bimbar]

You will have to use one firewall rule for each of the uplinks with the appropriate source prefix and then configure the right gateway in the firewall rule.

If those prefixes are dynamic, right now you're out of luck.

[jbattermann]

You will have to use one firewall rule for each of the uplinks with the appropriate source prefix and then configure the right gateway in the firewall rule.

If those prefixes are dynamic, right now you're out of luck.

I have tried that @bimbar but that alone did not work, unfortunately. Clients still always picked the default gateway (which is WAN1, not the policy-routing target, WAN2/GW_WAN2 one) - which I assume makes sense.. how would they know, based on firewall rules, which route (wanting) to go out... RA wise they get IPs assigned out of both IPv6 subnets but what I am missing is how can clients know that for XYZ target networks, please use the non-default / WAN2 gateway?

Once that's clear/known to clients, I guess further FW rules make total sense (allowing the two lan clients' /64 subnets to access to/through the FW and also use the corresponding WAN gateway for that outwards) but my logical disconnect is, how would clients ever know which route to go? Setting up IPv6 FW rules has no effect on any routing 'announcements' or knowledge , at least not by default as far as I understand...

[bimbar]

You can set the gateway to use in the firewall rule, as a sort of policy routing.