CARP + IPv6 Router advertisements

[pv2b]

Hi.

I'm trying to set up a pair of OPNsense boxes in a CARP HA setup, with IPv6, and I'm stuck on how to get router advertisements to work correctly.

Router advertisements are set up as follows:

Router Advertisements: Stateless
Router Priority: Normal (Low on the secondary unit)
RA Interface: LAN_VIP11 (corresponds to the CARP address)
Advertize default gateway: Yes
DNS Servers: Set to the CARP address
Minimum interval: 200
Maximum interval: 600

Rest is left blank.

At this point I'm expecting router advertisements to advertise the CARP address as a default gateway to IPv6 clients on the lan, but instead, the link local addresses as well as the individual IP addresses of each router are all advertised as defautl gateways, causing CARP not to operate correctly. Sometimes it doesn't work right and traffic goes out of the secondary firewall, which means return traffic is never passed because triangular routing.

Am I missing some secret sauce to get CARP addresses to work properly with router advertisements?

[hbc]

IPv6 and RA usually use local-link addresses for routers. That is correct. Usually you even would not need CARP for default gateway redundacy, since clients collect RA from all routers and those that are on-link and store at least two for use.

I just checked my servers and they just have the link-local adresses of both carp members, but no other global ipv6 addresses.

To limit the storage needed for the Default Router List, a host MAY choose not to store all of the router addresses discovered via advertisements.  However, a host MUST retain at least two router addresses and SHOULD retain more.

https://tools.ietf.org/html/rfc4861

In a test where the backup device was used as gateway, there were no issues, since states from primary gateway were sync'ed via pfsync. So asynchrone routing should not be any issue if cabeling is correct.

BTW: RA interface option does not mean that the vip ip is advertised as gateway. It is just the interface ra daemon should bind to and generate RA for and IMHO it is just a (useless) cosmetical thing in gui. If you have a look at the generated radvd.conf in /var/etc,  you will not see any vip interface, just the physical one and it does not make any difference if you select OPT directly or OPT_vip. The generated configuration will not change.

Edit: Correction -> changes are made

Code: [Select]

--- radvd.conf-novip    2020-02-27 16:25:32.993687000 +0100
+++ radvd.conf-vip      2020-02-27 16:24:50.078941000 +0100
@@ -262,11 +262,14 @@
        AdvManagedFlag on;
        AdvOtherConfigFlag on;
        prefix 2002:db0::/64 {
-               DeprecatePrefix on;
+               DeprecatePrefix off;
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr on;
        };
+       route ::/0 {
+               RemoveRoute off;
+       };

[pv2b]

Okay, but then I run into a problem. Because my upstream router will route to the primary firewall using its CARP address (I have control over it and I can change the configuration need be), there's a possibility of triangular routing, where for example the secondary firewall receives a packet to be sent outwards, and the nteh priamry firewall receives the response, which it will then drop because there's no corresponding state in the firewall.

pfsync is a thing, yes (and I have it implemented), but because it's asynchronous, the way it works in opnsense it's not usable for active-active scenarios like that, rather useful as a mechanism to stop long-running connections from dropping on failover.

Or at least the above is what I understand. Is any of this wrong?

[hbc]

I see your problem. I think my test ping6 just works, because icmp is stateless anyway and needs seperate rules in/out. So it does not matter which fw is passed on return.

Did you try RA HIGH in primary and RA LOWon backup as RA priority? Instead normal and low?

[pv2b]

I tried again with HIGH and LOW. Still broken. At least from my windows machine on the LAN. Request packet goes to the secondary firewall, goes out, reply packet is handled by primary firewall.

At this point I have no idea how you're supposed to get CARP to work with IPv6 in the current state of OPNsense, other than that it's apparently possible. At least not if you actually want to use strict state.

What do other vendors do here? I mean I would expect that the resonable thing would be to work analogously to how it works with IPv4, only sending RA for the CARP MAC and VIP.

[hbc]

I just found this for pfSense. Same problem there: https://forum.netgate.com/topic/133306/ipv6-carp-vip-with-route-advertisements/8

And I just checked my pfSense cluster end there, the radv-daemon is inactive on backup node. It only gets active on failover.

Edit:
I have to correct myself. The OPT_vip changes settings. Here is the diff:

Code: [Select]

--- radvd.conf-novip    2020-02-27 16:25:32.993687000 +0100
+++ radvd.conf-vip      2020-02-27 16:24:50.078941000 +0100
@@ -262,11 +262,14 @@
        AdvManagedFlag on;
        AdvOtherConfigFlag on;
        prefix 2002:db0::/64 {
-               DeprecatePrefix on;
+               DeprecatePrefix off;
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr on;
        };
+       route ::/0 {
+               RemoveRoute off;
+       };

[pv2b]

Oh! So the way it's supposed to work is that radvd is supposed to be only running on the primary node.

That doesn't seem to be the case on my cluster at least.

[hbc]

It's the way it works on pfSense. On OPNsense my radv-daemons are also running on both nodes.

I will try to disable radv-service on my backup node and use the carp trigger to start it on failover. The question is how to configure the service not to auto-start on boot. Script will take care of backup run state.

https://docs.opnsense.org/development/backend/autorun.html
https://docs.opnsense.org/development/backend/carp.html

Edit:

I created a script based on the 50-frr script, that starts radvd on MASTER and stops on BACKUP:

/usr/local/etc/rc.syshook.d/carp/20-radvd

Code: [Select]

#!/usr/local/bin/php

<?phprequire_once('config.inc');require_once('util.inc');$subsystem = !empty($argv[1]) ? $argv[1] : '';$type = !empty($argv[2]) ? $argv[2] : '';if ($type != 'MASTER' && $type != 'BACKUP') {    log_error("Carp '$type' event unknown from source '{$subsystem}'");    exit(1);}if (!strstr($subsystem, '@')) {    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");    exit(1);}switch ($type) {    case 'MASTER':        shell_exec('/usr/local/sbin/pluginctl -s radvd start');        break;    case 'BACKUP':        shell_exec('/usr/local/sbin/pluginctl -s radvd stop');        break;}

So radvd just runs on the active node, only one ipv6 gateway is advertised, no async routing. The downside of this 'hack':
In case of failover, it needs time to failover, start radvd on BACKUP node, advertise it to clients and clients have to start using new gateway. Have to test how long this will need.