Messages

Absolutely, I'm also aware of this issue and was already somewhat surprised that it worked instantly... We will see if I encounter more serious issues down-the-road, but for now most of my IPv6 network runs at least.

Now I started running into issues with Docker and IPv6 (due to no NAT the source-address being masked), which are also just painful - a topic for another day...

[only]

FYI... I just decided to use a workaround of a more loose "Allow IPv6 Internet" firewall rule, which is now still bound to the LAN interface, but does not actually check the sources address anymore, like under IPv4. I know this is not perfect, but this allows the clients to randomly choose an IPv6 address, prefix and gateway and to more equally distribute across the OPNsense instances.

One should keep in mind, that this only works here, as both OPNsense IPv6 prefix delegations are originating from the same /56 IPv6-prefix. With different IPv6-prefixes, one would indeed need to create a locally administered prefix and perform e.g. NPT with RA-CARPs...

Well... Crap. Before discarding this setup, here are my final questions:

• RFCs are being mentioned here multiple times, can someone call one specific out?
• By any chance, does anybody know which program is receiving the RAs under Linux? As the problem is present under Android and Debian, I suspect the kernel itself, which is somewhat odd... At least this would be a start for a proper fix.
• Similar to before: By any chance some ideas of how to add the src-addr to the Linux IPv6 route table for multipath route-entries? Which program manages these?
• I can't be the first one stumbling into this... Any ideas for keywords to look for? Maybe you also found some resource on that before? Anything to get a more clear view on this issue and potentially being able to get hold of the right persons...

Here is an excerpt for the IPv6 routing table (via "ip -6 route"), which suspiciously uses multipath routing, but these entries do not specify the proper source-address... But this is working fine for the other entries, so the selection is working well there?

Code Select Expand

2003:f5:7f19:954a::/64 dev enp5s0f0 proto ra metric 100 pref medium
2003:f5:7f19:958a::/64 dev enp5s0f0 proto ra metric 100 pref medium
2003:f5:7f2f:924a::2000 dev enp5s0f0 proto kernel metric 100 pref medium
2003:f5:7f2f:924a::/64 dev enp5s0f0 proto ra metric 100 pref medium
2003:f5:7f2f:928a::/64 dev enp5s0f0 proto ra metric 100 pref medium
fe80::/64 dev tun42 proto kernel metric 256 pref medium
fe80::/64 dev enp5s0f0 proto kernel metric 1024 pref medium
default proto ra metric 100 pref medium
nexthop via fe80::62be:b4ff:fe1f:7878 dev enp5s0f0 weight 1
nexthop via fe80::62be:b4ff:fe1d:f824 dev enp5s0f0 weight 1

Thank you very much for all your input! This at least reassures me that I'm somewhat not the idiot here :P

The concept of a HA cluster installation of OPNsense is to look like a single device to all clients for both IPv4 and IPv6. This is what works. What you are trying to build is something I have never done, is not in the docs, so you are on your own treading new paths here.

Indeed, I want to use dynamic IPv6 prefixes, therefore I cannot hard-code both OPNsense instances to advertise the same IPv6 prefix, for which then a IPv6 CARP gateway would make sense.

you are on your own treading new paths here.

Possibly there is a bug.

From my understanding, OPNsense is not really doing anything wrong here... The HA-aspect is in reality even not applicable here, as it only synchronizes the config and the conntrack-states (...), which are not share-able with this setup.

I understand now that you want to advertise two different prefixes with two different routers. But that is not "OPNsense HA" as devised by the HA implementation. OPNsense HA implies CARP and failover for everything. Regardless of the protocol.

Indeed. You are absolutely right. I have updated the post title accordingly... Any further ideas? I really want to avoid escalating this into the kernel mailing lists, as they are always being spammed too much by mailinglist-address-scrapers :P

@Patrick Hmm, this does not sound right as we are talking about IPv6 here and the two OPNsense instances are receiving two *distinct* IPv6 prefixes (due to design derived from same /56 prefix). Yes, for IPv4 I fully understand (and implemented) CARP addresses for the IPv4 gateway address.

Instead, with this setup, the clients should be able to use both IPv6 prefixes at their own discretion. Furthermore, if you use CARP for the RA-source address, then the client can only use the IPv6 prefix of the current CARP-MASTER, as the traffic for the other one would be once again dropped as the BACKUPs prefix would be unknown to the MASTER - right? Hence, defeating the idea of simultaneous use of both IPv6 prefixes.

Maybe you can share some reference regarding the IPv6 CARP address use for Router Advertisements? Especially since that recommendation would then still not explain the issue I'm observing and be only a "fix" by breaking one set of derived IPv6 addresses for the clients.

[Setup]

Hi!

Long time lurker, first time poster. Let's get started by describing my setup, so the following noticed misbehavior makes more sense to you.

Setup

Uplink (Fritz!Box) -> OPNsense#1 -> LAN
                  \-> OPNsense#2 -> LAN

Both OPNsense instances are configured as HA via a dedicated RJ45 connection, and #1 synchronizes to #2. The Fritz!Box is obtaining an /56 IPv6 prefix from the ISP and has prefix delegation enabled. This allows both OPNsense instances to obtain an /58 IPv6 prefix for their WAN interfaces. Let's add some more concrete number to this:

Code Select Expand

// Fritz!Box
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9500::/56 (two /57 are then derived for home-network and guest-network)

// OPNsense#1
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9580::/58
MAC in LAN: 60:be:b4:1d:f8:24
IPv6 link-local in LAN: fe80::62be:b4ff:fe1d:f824/64

// OPNsense#2
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9540::/58
MAC in LAN: 60:be:b4:1f:78:78
IPv6 link-local in LAN: fe80::62be:b4ff:fe1f:7878/64

Client

I have at least one client connected to the LAN network. In reality, multiple different Linux-based clients do exist, like Android, Linux Mint or Debian - all experience the issue. After connecting, this is the IPv6-configuration (I removed IPv4 here for simplification) of one Linux-machine (Linux 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux):

Code Select Expand

4: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:90:fa:e5:b6:9a brd ff:ff:ff:ff:ff:ff
    inet6 2003:f5:7f19:954a::2000/128 scope global dynamic noprefixroute
       valid_lft 5346sec preferred_lft 2646sec
    inet6 2003:f5:7f19:954a:4da2:82bb:4829:2cae/64 scope global temporary dynamic
       valid_lft 86119sec preferred_lft 14119sec
    inet6 2003:f5:7f19:954a:560e:ff4a:2078:76b0/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86119sec preferred_lft 14119sec
    inet6 2003:f5:7f19:958a:906e:97a2:c6e3:7560/64 scope global temporary dynamic
       valid_lft 86159sec preferred_lft 14159sec
    inet6 2003:f5:7f19:958a:7eba:6d14:9339:c9f4/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86159sec preferred_lft 14159sec
    inet6 fe80::6b62:2f6f:8e71:3de2/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

...so, we can see that the machine received the RAs from both OPNsense instances and has chosen two IPv6 addresses for each of them: One "temporary" and one "mngtmpaddr".

The Issue
Let's ping some IPv6 from the affected system:

Code Select Expand

ping6 2606:4700:4700::1111...whoops - no response! Sometimes the command does actually work, sometimes not. Reconnecting or waiting does help, until it stops working once again...

Tracing the Issue
Okay. Why? Let's take a look into the firewall logs from OPNsense#1:

Code Select Expand

__timestamp__    2026-01-01T22:44:11
action    [block]
anchorname   
class    0x00
dir    [in]
dst    2606:4700:4700::1111
hoplimit    64
interface    vlan04
ipversion    6
label    Custom reject / state violation rule
length    64
protoname    ipv6-icmp
protonum    58
reason    match
rid    86dbbe50f6310147e4cafeb1ed54d663
src    2003:f5:7f19:954a:4da2:82bb:4829:2cae
Hmm - the ping request got sent to the OPNsense#1 instance and got dropped there, as the "Allow IPv6 internet" rule did not match and got catched by my "Custom reject / state violation rule" (just logs and rejects any not matched package similar to the default block rule). Indeed, this is because of the src-address: 2003:f5:7f19:954a:4da2:82bb:4829:2cae, which is matching the IPv6 prefix of OPNsense #2!

So, the clients in question seem to randomly sent the traffic of their IPv6 source address from e.g. OPNsense#2 to the OPNsense#1 instance, which in turn drops the packets, as their do not originate from its interface IPv6 prefix.

Let's confirm, that the correct link-local addresses are part of the router-advertisements, so the clients know the correct MAC for the advertised IPv6-prefix. Filtering via "icmpv6.type == 134" in Wireshark upon connecting the client interface, two advertisements are received:

Code Select Expand

Ethernet II, Src: SBluetech_1d:f8:24 (60:be:b4:1d:f8:24), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1d:f824, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Prefix information : 2003:f5:7f19:958a::/64)

Ethernet II, Src: SBluetech_1f:78:78 (60:be:b4:1f:78:78), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1f:7878, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Prefix information : 2003:f5:7f19:954a::/64)

...so the prefix 2003:f5:7f19:958a::/64 should be associated with fe80::62be:b4ff:fe1f:7878. Let's check the IPv6 ICMP Neighbor Solicitation for that link-local address:

Code Select Expand

Ethernet II, Src: SBluetech_1f:78:78 (60:be:b4:1f:78:78), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1f:7878, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Source link-layer address : 60:be:b4:1f:78:78)
...so the link-local address fe80::62be:b4ff:fe1f:7878 is associated with 60:be:b4:1f:78:78.

Let's see if the ICMPv6 echo requests are also sent there:

Code Select Expand

Ethernet II, Src: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a), Dst: SBluetech_1f:78:78 (60:be:b4:1f:78:78)
Internet Protocol Version 6, Src: 2003:f5:7f19:958a:ea3b:e9b1:abf7:aa03, Dst: 2606:4700:4700::1111

Huh?! The client used its address from the 2003:f5:7f19:958a::/64 IPv6 prefix from OPNsense#1 to sent to the 60:be:b4:1f:78:78 MAC of OPNsense#2.

What?!
As far as I understand IPv6 networking, the client should have chosen the other destination MAC from OPNsense#1 for sending the package, so it would have arrived at the correct instance. The filtering rule(s) in OPNsense#1/#2 are correct, as their are restricting the source-addresses to originate from their advertised IPv6 prefixes and discard (wrong) addressed traffic before forwarding.

Some more thoughts?

• In these multi-gateway IPv6 networks, are all gateways required to serve all locally advertised IPv6 prefixes? No, this does not make sense, as they could differ and not every gateway would be able to forward traffic for all prefixes. Also, ICMPv6 Router Advertisements do specify the prefix length, hence the client can figure out which one to pick for its chosen address.
• Bug in Linux Kernel? Meh, IPv6 is old, that I doubt this is bug would have flown that long under the radar. Maybe something in higher layers? Like selection of binding sockets to addresses in TCP? But why would then also ICMP fail? Same src-addr selection logic used, but then really in the kernel?!
• Are there some RFCs requiring this behavior?
• Am I misunderstanding the resolution of MACs in IPv6? Are Route Advertisements and source-address selection really linked like this?

So yeah... I'm somewhat confused what I find here... Do you have any more ideas? Are there details for you to understand missing? Yes, I have not masked any IPv6-prefixes or MACs here intentionally, as maybe their specific values trigger this behavior?

Looking forward to some ideas :P