OPNsense HA (CARP) with IPv6, inconsistent Prefix Delegation from upstream

[opn_mndr12101]

G'day everyone.

I am running an OPNsense HA setup (CARP) with two nodes:

- opnsense-alpha
- opnsense-beta

Both are connected to the same upstream router, a FRITZ!Box, which receives an IPv6 prefix from the ISP.

_Upstream situation_

Both OPNsense nodes are connected to the same L2 segment (WAN side). The Fritz!Box receives a delegated prefix from the ISP:

2001:9e8:1484:3080::/58


_Observed behavior_

Each OPNsense node independently requests DHCPv6 Prefix Delegation. However, the Fritz!Box assigns different delegated prefixes to each node:

- opnsense-alpha: 2001:9e8:1484:3080::/58
- opnsense-beta:  2001:9e8:1484:30c0::/58

As a result, the tracked internal networks differ:

    alpha:
        LAN: 2001:9e8:1484:3081::/64
        DMZ: 2001:9e8:1484:3082::/64

    beta:
        LAN: 2001:9e8:1484:30c1::/64
        DMZ: 2001:9e8:1484:30c2::/64

_Problem_

This leads to a fundamental issue:
- The two HA nodes do not share the same IPv6 prefixes
- Internal networks differ depending on which node is active
- CARP failover results in a different IPv6 network for clients

This not just breaks the expectation of transparent failover, but also confuses v6 servers (and may be even clients?) running in a the DMZ.

_Question_

Is this expected behavior with DHCPv6-PD in an HA setup?

More specifically, is there any supported way in OPNsense to achieve consistent IPv6 prefixes across HA nodes when using DHCPv6-PD? Or is this fundamentally incompatible unless:
- a static routed prefix is used, or
- only one node performs DHCPv6-PD?

_Additional notes_
WAN connectivity and PD itself work fine on both nodes. Issue only appears when combining CARP, Track Interface and DHCPv6-PD from a consumer router (Fritz!Box). This is not about broken IPv6 connectivity, but about design limitations of DHCPv6-PD in HA scenarios.

Any guidance or recommended architecture would be appreciated, thank you!

[nero355]

IMHO :
- This is a silly HA setup : When the Fritz!Box goes down your whole network goes down too !!
- If you remove the Fritz!Box the issue would stay the same : Your ISP would also give you two different IPv6 Prefixes !!
At least a lot of them do AFAIK because they don't check if one Customer has requested more than one IPv6 Prefix.

This kind of HA setup is more suited for Business Customers that get a Static IPv4 Address or whole Subnet + Static IPv6 Prefix.

[Monviech (Cedrik)]

I run two opnsense behind my fritzbox in HA with an ndp proxy. That way both devices announce the same prefix all the time, doesn't matter which of them is online. The CARP hook prevents the ndp proxy from running on both nodes at the same time.

Please note that DHCPv6-PD might be better for most environments.

HA support was a natural consequence of the stateless design and the CARP hook from when I wrote the ndp proxy. Works pretty stable for me (in combination with a Fritzbox).

https://docs.opnsense.org/manual/ndp-proxy-go.html#high-availability

[opn_mndr12101]

This is the os-ndp-proxy-go plugin? Do I need the os-ndproxy, too?

[Monviech (Cedrik)]

You only need ndp-proxy-go.

The manual explains an example setup with it.

I would recommend to read it whole as ndp proxying is a different concept than DHCPv6-PD.

The magic in a HA environment is that both OPNsense learn the same on link /64 prefix from the RAs sent by the Fritzbox, and will then proxy that to all clients behind it. That means if there is a failover, the prefix will not change, its transparent.

There is no DHCPv6 on WAN needed, only SLAAC.

[Patrick M. Hausen]

Another option is SLAAC for the individual systems' uplinks but static configuration for all internal networks. But that only works if you do not have to deal with dynamic changing prefixes. I avoid these like the plague.

[opn_mndr12101]

The magic in a HA environment is that both OPNsense learn the same on link /64 prefix from the RAs sent by the Fritzbox...

I'm not getting it: Shouldn't I the get the /58 prefix at the OPNsense and slice it up into /64 there? Reading the documentation, I do not see, how these parts (Track Interface vs. ndp-proxy) fit together. This is more a question, if I really want this, not how to do it, I love reading man pages :-)

[Monviech (Cedrik)]

Thats the difference between PD and an ndp proxy.

An ndp proxy works without DHCPv6, there wont be tracked interfaces or your /58 prefix.

There will only be a single /64 prefix, the one your OPNsenes will both get via SLAAC from the Fritzbox.

Then this /64 will be propagated to all vlans behind the OPNsense, the proxy supports firewall aliases to split the same /64 into different security zones. There is no classic subnetting here (logical different prefixes per vlan), but firewall rules that target individual hosts in groups that have been dynamically learned by the proxy.

That is also why HA will work seamlessly, all devices will always have the exact same prefix and the same address, regardless if there are multiple OPNsense, since the Fritzbox will remain the Router Identity since it owns that /64 network. What happens when you delegate a prefix, is that the router identity changes to the router that received that prefix. And in your case you end up with multiple router identities and prefixes flap because of this. DHCPv6 assumes the next router to be a single logical instance, not multiple different ones.

If you want this or not depends on your personal preferences, this is pretty novel, a concept like this (for HA) did not really exist just a few months ago.

Reading the man pages is a good idea, I wrote them :)