OPNsense HA w/ IPv6 PD & Dual-Stack LAN Continuity Behind AT&T BGW320

[Wynbr00k]

OPNsense High Availability Behind AT&T BGW320
Preserving IPv6 Prefix Delegation (PD) and Dual‑Stack LAN Continuity

This document describes a working, tested method for running OPNsense 26.7.2 in a high-availability (HA) configuration behind an AT&T BGW320 gateway in passthrough mode, while maintaining:

• A stable IPv6 delegated prefix (IA-PD) across failover
• Consistent IPv6 SLAAC behavior
• Correct Router Advertisements (RA)
• dnsmasq SLAAC-glean AAAA/PTR synthesis
• Unbound recursion and local zones
• Dual-stack LAN continuity
• Seamless CARP failover and failback

The solution uses a single CARP syshook (30-wan-identity) to manage WAN identity, fencing, DHCPv6 behavior, radvd state, and IPv6 interface configuration.

----------------------------------------------------------------------
AT&T IPv6 PD Structure (Anonymized Example)
----------------------------------------------------------------------

AT&T assigns a /60:

    2600:1700:ffff:fff0::/60

Within this block:

• ...fff0::/64 — BGW LAN
• ...fff1::/64 through ...fff7::/64 — reserved
• ...fff8::/64 through ...ffff::/64 — delegated (highest handed out first)

With one IA_PD request, the delegated prefix is always:

    2600:1700:ffff:ffff::/64

This prefix is preserved across failover by ensuring only one firewall presents the WAN identity at any time.

----------------------------------------------------------------------
WAN Identity Model
----------------------------------------------------------------------

Both firewalls share:

• A virtual MAC (vMAC)
• A virtual DHCPv6 DUID (vDUID)

Only the CARP MASTER presents these values.
The BACKUP removes them and disables WAN IPv6 entirely.

This ensures AT&T always assigns the PD to the active MASTER.

----------------------------------------------------------------------
LAN IPv6 Model
----------------------------------------------------------------------

LAN IPv6 is provided via CARP VIPs:

• IPv6 link-local VIP (RA source)
• IPv6 GUA VIP (RDNSS + DNS AAAA)

dnsmasq provides:
• DHCPv4
• SLAAC gleaning (AAAA + PTR) (requires 'empty' DHCPv6 Range Configuration to enable gleaning)
• Local forward/reverse zones

radvd provides:
• Router Advertisements
• Prefix
• Gateway
• RDNSS

unbound provides:
• Recursion
• Local zones
• Reverse zones

dnsmasq RA lifetime is set to 0 to disable dnsmasq RA while preserving SLAAC gleaning.

----------------------------------------------------------------------
Failover Logic (Role-Accurate)
----------------------------------------------------------------------

MASTER → BACKUP

Box becoming BACKUP:
• Stops dhcp6c
• Clears vMAC
• Disables WAN IPv6
• Deletes vDUID
• Disables radvd
• Loses PD
• Becomes passive

Box becoming MASTER:
• Stops any lingering dhcp6c
• Performs fencing
• Restores vMAC
• Restores vDUID
• Enables WAN IPv6 (DHCPv6)
• Starts dhcp6c
• Enables radvd
• Gains PD
• Becomes active

AT&T assigns the PD to the box that becomes MASTER.

BACKUP → MASTER

Box becoming MASTER:
• Stops any lingering dhcp6c
• Performs fencing
• Restores vMAC
• Restores vDUID
• Enables WAN IPv6 (DHCPv6)
• Starts dhcp6c
• Enables radvd
• Gains PD
• Becomes active

Box becoming BACKUP:
• Stops dhcp6c
• Clears vMAC
• Disables WAN IPv6
• Deletes vDUID
• Disables radvd
• Loses PD
• Becomes passive

LAN IPv6 continuity is preserved throughout.

----------------------------------------------------------------------
The Syshook
----------------------------------------------------------------------

A single CARP syshook (30-wan-identity) handles:

• CARP role detection
• Fencing (passive + active)
• DHCPv6 teardown/restore
• vMAC/vDUID management
• WAN IPv6 enable/disable
• radvd model-level control
• Retry logic

This script is the core of the HA IPv6 solution.

----------------------------------------------------------------------
Notes
----------------------------------------------------------------------

• No DHCPv6 is used on LAN (pure SLAAC, IPv6 Mode - None)
• No authoritative unbound zones are used
• Only one syshook is required
• Potential to extend to multiple internal interfaces/vLANs (not tested here)

ymmv

[Wynbr00k]

Had a slight issue arise on the reboot of the OPNsense v26.7.3 update. The native EUI-64 MAC-address derived local-link address was missing from the igc0 (LAN) interface. igc0 did have the CARP local-link address configured. This hardware local-link 'absence' caused Dnsmasq RA to collide with radvd RA since now both were advertising the same RA. The investigation into the missing native EUI-64 MAC derived local-link led to the following:

The actual FreeBSD kernel source appears to confirm the cause.

The relevant function is in6_ifattach() in sys/netinet6/in6_ifattach.c. Before it ever generates the MAC-derived (EUI-64) link-local address, it does this:

c
ia = in6ifa_ifpforlinklocal(ifp, 0);
if (ia == NULL) {
    error = in6_ifattach_linklocal(ifp, altifp);
} else
    ifa_free(&ia->ia_ifa);

If in6ifa_ifpforlinklocal() finds any link-local address already sitting on the interface, ia is non-NULL, and in6_ifattach_linklocal() — the code that would derive and install the native fe80:: from the interface's real MAC — never runs at all. The interface is simply left with whatever link-local it already had.

The key detail is in in6ifa_ifpforlinklocal() itself, in in6.c:

c
struct in6_ifaddr *
in6ifa_ifpforlinklocal(struct ifnet *ifp, int ignoreflags)
{
    ...
    TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) {
        if (ifa->ifa_addr->sa_family != AF_INET6)
            continue;
        if (IN6_IS_ADDR_LINKLOCAL(IFA_IN6(ifa))) {
            if ((((struct in6_ifaddr *)ifa)->ia6_flags & ignoreflags) != 0)
                continue;
            ifa_ref(ifa);
            break;
        }
    }
    ...
}

in6_ifattach() calls this with ignoreflags = 0 — meaning nothing is excluded. It matches the first fe80:: it finds on the interface's address list, full stop, with no check for how that address got there. A CARP-assigned vhid link-local is just another entry on ifp->if_addrhead as far as this loop is concerned — it's completely indistinguishable from a "real" autoconfigured one.

And per FreeBSD's own Developer's Handbook, this generation step runs "when the interface becomes up (IFF_UP)" — i.e., every time the interface transitions to up, not just once at boot. That lines up exactly with what was experienced: the manual bounce (delete CARP's LL, ifconfig igc0 down / up) hit that check with genuinely zero fe80:: addresses present, so the native one finally got generated — and once both existed, nothing ever re-runs the check to disturb either one, which is why re-adding the CARP VIP alongside it afterward 'appears' safe (until the next native/CARP 'race occurs w/ a clear ifp).

So, the most likely causal chain: OPNsense brings up igc0 at boot → if CARP's kernel module manages to attach its own fe80:: vhid address to that interface before/at the same up-transition that would otherwise trigger native LL generation → in6ifa_ifpforlinklocal(ifp, 0) finds CARP's address first → native EUI-64 generation is skipped entirely for that up-event, with nothing to retrigger it short of another down/up cycle starting from zero fe80:: addresses.

An actual VIP IP Alias for the native MAC local-link has been configured to hopefully eliminate any further FreeBSD/CARP 'race' condition.

Sidenote: this issue was seen only on the machine with Intel hardware/drivers and not on the machine with Realtek hardware/drivers, but the machine specific VIP IP Alias was configured on both 'just to be safe'. Also, not sure how the RFC is worded/interpreted, as to whether FreeBSD should be decision testing off the local-link it is trying to configure and not just 'any' local-link configured at the time or is OPNsense preempting some function/configuration it should not.

[Wynbr00k]

knew i should have double/triple checked, old FreeBSD mirror; this from:

Fetched directly from raw.githubusercontent.com/freebsd/freebsd-src/releng/15.1/sys/netinet6/ (that branch is exactly where the p3 errata patches land, so this is the correct source):

c
// in6_ifattach.c, in6_ifattach():
NET_EPOCH_ENTER(et);
ia = in6ifa_ifpforlinklocal(ifp, 0);
NET_EPOCH_EXIT(et);
if (ia == NULL)
    in6_ifattach_linklocal(ifp, altifp);
else
    ifa_free(&ia->ia_ifa);
c
// in6.c, in6ifa_ifpforlinklocal():
CK_STAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) {
    if (ifa->ifa_addr->sa_family != AF_INET6)
        continue;
    if (IN6_IS_ADDR_LINKLOCAL(IFA_IN6(ifa))) {
        if ((((struct in6_ifaddr *)ifa)->ia6_flags & ignoreflags) != 0)
            continue;
        ifa_ref(ifa);
        break;
    }
}

Same gating logic.