Messages

The concept of a HA cluster installation of OPNsense is to look like a single device to all clients for both IPv4 and IPv6. This is what works. What you are trying to build is something I have never done, is not in the docs, so you are on your own treading new paths here.

Indeed, I want to use dynamic IPv6 prefixes, therefore I cannot hard-code both OPNsense instances to advertise the same IPv6 prefix, for which then a IPv6 CARP gateway would make sense.

you are on your own treading new paths here.

Possibly there is a bug.

From my understanding, OPNsense is not really doing anything wrong here... The HA-aspect is in reality even not applicable here, as it only synchronizes the config and the conntrack-states (...), which are not share-able with this setup.

I understand now that you want to advertise two different prefixes with two different routers. But that is not "OPNsense HA" as devised by the HA implementation. OPNsense HA implies CARP and failover for everything. Regardless of the protocol.

Indeed. You are absolutely right. I have updated the post title accordingly... Any further ideas? I really want to avoid escalating this into the kernel mailing lists, as they are always being spammed too much by mailinglist-address-scrapers :P

@Patrick Hmm, this does not sound right as we are talking about IPv6 here and the two OPNsense instances are receiving two *distinct* IPv6 prefixes (due to design derived from same /56 prefix). Yes, for IPv4 I fully understand (and implemented) CARP addresses for the IPv4 gateway address.

Instead, with this setup, the clients should be able to use both IPv6 prefixes at their own discretion. Furthermore, if you use CARP for the RA-source address, then the client can only use the IPv6 prefix of the current CARP-MASTER, as the traffic for the other one would be once again dropped as the BACKUPs prefix would be unknown to the MASTER - right? Hence, defeating the idea of simultaneous use of both IPv6 prefixes.

Maybe you can share some reference regarding the IPv6 CARP address use for Router Advertisements? Especially since that recommendation would then still not explain the issue I'm observing and be only a "fix" by breaking one set of derived IPv6 addresses for the clients.

[Setup]

Hi!

Long time lurker, first time poster. Let's get started by describing my setup, so the following noticed misbehavior makes more sense to you.

Setup

Uplink (Fritz!Box) -> OPNsense#1 -> LAN
                  \-> OPNsense#2 -> LAN

Both OPNsense instances are configured as HA via a dedicated RJ45 connection, and #1 synchronizes to #2. The Fritz!Box is obtaining an /56 IPv6 prefix from the ISP and has prefix delegation enabled. This allows both OPNsense instances to obtain an /58 IPv6 prefix for their WAN interfaces. Let's add some more concrete number to this:

Code Select Expand

// Fritz!Box
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9500::/56 (two /57 are then derived for home-network and guest-network)

// OPNsense#1
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9580::/58
MAC in LAN: 60:be:b4:1d:f8:24
IPv6 link-local in LAN: fe80::62be:b4ff:fe1d:f824/64

// OPNsense#2
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9540::/58
MAC in LAN: 60:be:b4:1f:78:78
IPv6 link-local in LAN: fe80::62be:b4ff:fe1f:7878/64

Client

I have at least one client connected to the LAN network. In reality, multiple different Linux-based clients do exist, like Android, Linux Mint or Debian - all experience the issue. After connecting, this is the IPv6-configuration (I removed IPv4 here for simplification) of one Linux-machine (Linux 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux):

Code Select Expand

4: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:90:fa:e5:b6:9a brd ff:ff:ff:ff:ff:ff
    inet6 2003:f5:7f19:954a::2000/128 scope global dynamic noprefixroute
       valid_lft 5346sec preferred_lft 2646sec
    inet6 2003:f5:7f19:954a:4da2:82bb:4829:2cae/64 scope global temporary dynamic
       valid_lft 86119sec preferred_lft 14119sec
    inet6 2003:f5:7f19:954a:560e:ff4a:2078:76b0/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86119sec preferred_lft 14119sec
    inet6 2003:f5:7f19:958a:906e:97a2:c6e3:7560/64 scope global temporary dynamic
       valid_lft 86159sec preferred_lft 14159sec
    inet6 2003:f5:7f19:958a:7eba:6d14:9339:c9f4/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86159sec preferred_lft 14159sec
    inet6 fe80::6b62:2f6f:8e71:3de2/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

...so, we can see that the machine received the RAs from both OPNsense instances and has chosen two IPv6 addresses for each of them: One "temporary" and one "mngtmpaddr".

The Issue
Let's ping some IPv6 from the affected system:

Code Select Expand

ping6 2606:4700:4700::1111...whoops - no response! Sometimes the command does actually work, sometimes not. Reconnecting or waiting does help, until it stops working once again...

Tracing the Issue
Okay. Why? Let's take a look into the firewall logs from OPNsense#1:

Code Select Expand

__timestamp__    2026-01-01T22:44:11
action    [block]
anchorname   
class    0x00
dir    [in]
dst    2606:4700:4700::1111
hoplimit    64
interface    vlan04
ipversion    6
label    Custom reject / state violation rule
length    64
protoname    ipv6-icmp
protonum    58
reason    match
rid    86dbbe50f6310147e4cafeb1ed54d663
src    2003:f5:7f19:954a:4da2:82bb:4829:2cae
Hmm - the ping request got sent to the OPNsense#1 instance and got dropped there, as the "Allow IPv6 internet" rule did not match and got catched by my "Custom reject / state violation rule" (just logs and rejects any not matched package similar to the default block rule). Indeed, this is because of the src-address: 2003:f5:7f19:954a:4da2:82bb:4829:2cae, which is matching the IPv6 prefix of OPNsense #2!

So, the clients in question seem to randomly sent the traffic of their IPv6 source address from e.g. OPNsense#2 to the OPNsense#1 instance, which in turn drops the packets, as their do not originate from its interface IPv6 prefix.

Let's confirm, that the correct link-local addresses are part of the router-advertisements, so the clients know the correct MAC for the advertised IPv6-prefix. Filtering via "icmpv6.type == 134" in Wireshark upon connecting the client interface, two advertisements are received:

Code Select Expand

Ethernet II, Src: SBluetech_1d:f8:24 (60:be:b4:1d:f8:24), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1d:f824, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Prefix information : 2003:f5:7f19:958a::/64)

Ethernet II, Src: SBluetech_1f:78:78 (60:be:b4:1f:78:78), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1f:7878, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Prefix information : 2003:f5:7f19:954a::/64)

...so the prefix 2003:f5:7f19:958a::/64 should be associated with fe80::62be:b4ff:fe1f:7878. Let's check the IPv6 ICMP Neighbor Solicitation for that link-local address:

Code Select Expand

Ethernet II, Src: SBluetech_1f:78:78 (60:be:b4:1f:78:78), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1f:7878, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Source link-layer address : 60:be:b4:1f:78:78)
...so the link-local address fe80::62be:b4ff:fe1f:7878 is associated with 60:be:b4:1f:78:78.

Let's see if the ICMPv6 echo requests are also sent there:

Code Select Expand

Ethernet II, Src: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a), Dst: SBluetech_1f:78:78 (60:be:b4:1f:78:78)
Internet Protocol Version 6, Src: 2003:f5:7f19:958a:ea3b:e9b1:abf7:aa03, Dst: 2606:4700:4700::1111

Huh?! The client used its address from the 2003:f5:7f19:958a::/64 IPv6 prefix from OPNsense#1 to sent to the 60:be:b4:1f:78:78 MAC of OPNsense#2.

What?!
As far as I understand IPv6 networking, the client should have chosen the other destination MAC from OPNsense#1 for sending the package, so it would have arrived at the correct instance. The filtering rule(s) in OPNsense#1/#2 are correct, as their are restricting the source-addresses to originate from their advertised IPv6 prefixes and discard (wrong) addressed traffic before forwarding.

Some more thoughts?

• In these multi-gateway IPv6 networks, are all gateways required to serve all locally advertised IPv6 prefixes? No, this does not make sense, as they could differ and not every gateway would be able to forward traffic for all prefixes. Also, ICMPv6 Router Advertisements do specify the prefix length, hence the client can figure out which one to pick for its chosen address.
• Bug in Linux Kernel? Meh, IPv6 is old, that I doubt this is bug would have flown that long under the radar. Maybe something in higher layers? Like selection of binding sockets to addresses in TCP? But why would then also ICMP fail? Same src-addr selection logic used, but then really in the kernel?!
• Are there some RFCs requiring this behavior?
• Am I misunderstanding the resolution of MACs in IPv6? Are Route Advertisements and source-address selection really linked like this?

So yeah... I'm somewhat confused what I find here... Do you have any more ideas? Are there details for you to understand missing? Yes, I have not masked any IPv6-prefixes or MACs here intentionally, as maybe their specific values trigger this behavior?

Looking forward to some ideas :P