Hi!
Long time lurker, first time poster. Let's get started by describing my setup, so the following noticed misbehavior makes more sense to you.
SetupUplink (Fritz!Box) -> OPNsense#1 -> LAN
\-> OPNsense#2 -> LAN
Both OPNsense instances are configured as HA via a dedicated RJ45 connection, and #1 synchronizes to #2. The Fritz!Box is obtaining an /56 IPv6 prefix from the ISP and has prefix delegation enabled. This allows both OPNsense instances to obtain an /58 IPv6 prefix for their WAN interfaces. Let's add some more concrete number to this:
// Fritz!Box
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9500::/56 (two /57 are then derived for home-network and guest-network)
// OPNsense#1
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9580::/58
MAC in LAN: 60:be:b4:1d:f8:24
IPv6 link-local in LAN: fe80::62be:b4ff:fe1d:f824/64
// OPNsense#2
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9540::/58
MAC in LAN: 60:be:b4:1f:78:78
IPv6 link-local in LAN: fe80::62be:b4ff:fe1f:7878/64
ClientI have at least one client connected to the LAN network. In reality, multiple different Linux-based clients do exist, like Android, Linux Mint or Debian - all experience the issue. After connecting, this is the IPv6-configuration (I removed IPv4 here for simplification) of one Linux-machine (Linux 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux):
4: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:90:fa:e5:b6:9a brd ff:ff:ff:ff:ff:ff
inet6 2003:f5:7f19:954a::2000/128 scope global dynamic noprefixroute
valid_lft 5346sec preferred_lft 2646sec
inet6 2003:f5:7f19:954a:4da2:82bb:4829:2cae/64 scope global temporary dynamic
valid_lft 86119sec preferred_lft 14119sec
inet6 2003:f5:7f19:954a:560e:ff4a:2078:76b0/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 86119sec preferred_lft 14119sec
inet6 2003:f5:7f19:958a:906e:97a2:c6e3:7560/64 scope global temporary dynamic
valid_lft 86159sec preferred_lft 14159sec
inet6 2003:f5:7f19:958a:7eba:6d14:9339:c9f4/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 86159sec preferred_lft 14159sec
inet6 fe80::6b62:2f6f:8e71:3de2/64 scope link noprefixroute
valid_lft forever preferred_lft forever
...so, we can see that the machine received the RAs from both OPNsense instances and has chosen two IPv6 addresses for each of them: One "temporary" and one "mngtmpaddr".
The IssueLet's ping some IPv6 from the affected system:
ping6 2606:4700:4700::1111...whoops - no response! Sometimes the command does actually work, sometimes not. Reconnecting or waiting does help, until it stops working once again...
Tracing the IssueOkay. Why? Let's take a look into the firewall logs from OPNsense#1:
__timestamp__ 2026-01-01T22:44:11
action [block]
anchorname
class 0x00
dir [in]
dst 2606:4700:4700::1111
hoplimit 64
interface vlan04
ipversion 6
label Custom reject / state violation rule
length 64
protoname ipv6-icmp
protonum 58
reason match
rid 86dbbe50f6310147e4cafeb1ed54d663
src 2003:f5:7f19:954a:4da2:82bb:4829:2cae
Hmm - the ping request got sent to the OPNsense#1 instance and got dropped there, as the "Allow IPv6 internet" rule did not match and got catched by my "Custom reject / state violation rule" (just logs and rejects any not matched package similar to the default block rule). Indeed, this is because of the src-address: 2003:f5:7f19:954a:4da2:82bb:4829:2cae,
which is matching the IPv6 prefix of OPNsense #2!So, the clients in question seem to randomly sent the traffic of their IPv6 source address from e.g. OPNsense#2 to the OPNsense#1 instance, which in turn drops the packets, as their do not originate from its interface IPv6 prefix.
Let's confirm, that the correct link-local addresses are part of the router-advertisements, so the clients know the correct MAC for the advertised IPv6-prefix. Filtering via "icmpv6.type == 134" in Wireshark upon connecting the client interface, two advertisements are received:
Ethernet II, Src: SBluetech_1d:f8:24 (60:be:b4:1d:f8:24), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1d:f824, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Prefix information : 2003:f5:7f19:958a::/64)
Ethernet II, Src: SBluetech_1f:78:78 (60:be:b4:1f:78:78), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1f:7878, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Prefix information : 2003:f5:7f19:954a::/64)
...so the prefix 2003:f5:7f19:958a::/64 should be associated with fe80::62be:b4ff:fe1f:7878. Let's check the IPv6 ICMP Neighbor Solicitation for that link-local address:
Ethernet II, Src: SBluetech_1f:78:78 (60:be:b4:1f:78:78), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1f:7878, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Source link-layer address : 60:be:b4:1f:78:78)
...so the link-local address fe80::62be:b4ff:fe1f:7878 is associated with 60:be:b4:1f:78:78.
Let's see if the ICMPv6 echo requests are also sent there:
Ethernet II, Src: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a), Dst: SBluetech_1f:78:78 (60:be:b4:1f:78:78)
Internet Protocol Version 6, Src: 2003:f5:7f19:958a:ea3b:e9b1:abf7:aa03, Dst: 2606:4700:4700::1111
Huh?! The client used its address from the 2003:f5:7f19:958a::/64 IPv6 prefix from OPNsense#1 to sent to the 60:be:b4:1f:78:78 MAC of OPNsense#2.
What?!As far as I understand IPv6 networking, the client should have chosen the other destination MAC from OPNsense#1 for sending the package, so it would have arrived at the correct instance. The filtering rule(s) in OPNsense#1/#2 are correct, as their are restricting the source-addresses to originate from their advertised IPv6 prefixes and discard (wrong) addressed traffic before forwarding.
Some more thoughts?
- In these multi-gateway IPv6 networks, are all gateways required to serve all locally advertised IPv6 prefixes? No, this does not make sense, as they could differ and not every gateway would be able to forward traffic for all prefixes. Also, ICMPv6 Router Advertisements do specify the prefix length, hence the client can figure out which one to pick for its chosen address.
- Bug in Linux Kernel? Meh, IPv6 is old, that I doubt this is bug would have flown that long under the radar. Maybe something in higher layers? Like selection of binding sockets to addresses in TCP? But why would then also ICMP fail? Same src-addr selection logic used, but then really in the kernel?!
- Are there some RFCs requiring this behavior?
- Am I misunderstanding the resolution of MACs in IPv6? Are Route Advertisements and source-address selection really linked like this?
So yeah... I'm somewhat confused what I find here... Do you have any more ideas? Are there details for you to understand missing? Yes, I have not masked any IPv6-prefixes or MACs here intentionally, as maybe their specific values trigger this behavior?
Looking forward to some ideas :P
In an HA cluster you are supposed to create a CARP address on LAN, preferably link-local. I use for example fe80::<vlan-id> on all networks. Then use that address as the source for your router advertisements. Works rock solid.
@Patrick Hmm, this does not sound right as we are talking about IPv6 here and the two OPNsense instances are receiving two *distinct* IPv6 prefixes (due to design derived from same /56 prefix). Yes, for IPv4 I fully understand (and implemented) CARP addresses for the IPv4 gateway address.
Instead, with this setup, the clients should be able to use both IPv6 prefixes at their own discretion. Furthermore, if you use CARP for the RA-source address, then the client can only use the IPv6 prefix of the current CARP-MASTER, as the traffic for the other one would be once again dropped as the BACKUPs prefix would be unknown to the MASTER - right? Hence, defeating the idea of simultaneous use of both IPv6 prefixes.
Maybe you can share some reference regarding the IPv6 CARP address use for Router Advertisements? Especially since that recommendation would then still not explain the issue I'm observing and be only a "fix" by breaking one set of derived IPv6 addresses for the clients.
The concept of a HA cluster installation of OPNsense is to look like a single device to all clients for both IPv4 and IPv6. This is what works. What you are trying to build is something I have never done, is not in the docs, so you are on your own treading new paths here.
Possibly there is a bug. I understand now that you want to advertise two different prefixes with two different routers. But that is not "OPNsense HA" as devised by the HA implementation. OPNsense HA implies CARP and failover for everything. Regardless of the protocol.
Quote from: Patrick M. Hausen on January 01, 2026, 11:40:20 PMThe concept of a HA cluster installation of OPNsense is to look like a single device to all clients for both IPv4 and IPv6. This is what works. What you are trying to build is something I have never done, is not in the docs, so you are on your own treading new paths here.
Indeed, I want to use dynamic IPv6 prefixes, therefore I cannot hard-code both OPNsense instances to advertise the same IPv6 prefix, for which then a IPv6 CARP gateway would make sense.
Quote from: Patrick M. Hausen on January 01, 2026, 11:40:20 PMyou are on your own treading new paths here.
Possibly there is a bug.
From my understanding, OPNsense is not really doing anything wrong here... The HA-aspect is in reality even not applicable here, as it only synchronizes the config and the conntrack-states (...), which are not share-able with this setup.
Quote from: Patrick M. Hausen on January 01, 2026, 11:40:20 PMI understand now that you want to advertise two different prefixes with two different routers. But that is not "OPNsense HA" as devised by the HA implementation. OPNsense HA implies CARP and failover for everything. Regardless of the protocol.
Indeed. You are absolutely right. I have updated the post title accordingly... Any further ideas? I really want to avoid escalating this into the kernel mailing lists, as they are always being spammed too much by mailinglist-address-scrapers :P
This is simply unsupported in the real world. You can't have multiple default routers which advertise different SLAAC prefixes in the same LAN.
While hosts theoretically SHOULD use the router which matches the selected source address, pretty much no OS does this. It's not just a Linux issue.
Cheers
Maurice
Quote from: Maurice on January 02, 2026, 01:27:15 AMThis is simply unsupported in the real world. You can't have multiple default routers which advertise different SLAAC prefixes in the same LAN.
I noticed that "in the real world" but still this comes as quite the surprise to me. Because all the IPng/IPv6 fundamentals textbooks I read more than a decade ago said that is how it's supposed to work. Multihoming - solved. An arbitrary number of addresses/prefixes on every host interface plus router advertisements will just do the right thing.
So like the OP I took it as a given that this setup should "just work".
But then again at some point in time host to host IPsec was mandatory for any compliant IPv6 implementation, too ...
Quote from: Patrick M. Hausen on January 02, 2026, 01:32:52 AMBecause all the IPng/IPv6 fundamentals textbooks I read more than a decade ago said that is how it's supposed to work.
Indeed, but unfortunately it doesn't. And not just for the "multiple default routers" use case. Another example which doesn't work in the real world are routers which don't advertise themselves as a default router (Router Lifetime 0), but do advertise more specific routes. Most clients ignore these and don't even bother to add them to their routing table. They just send everything to the default router.
Quote from: Patrick M. Hausen on January 02, 2026, 01:32:52 AMMultihoming - solved.
Multihoming is a beautiful thought experiment specified in many RFCs, plenty of which are ignored by real-world implementations...
Quote from: Maurice on January 02, 2026, 01:54:15 AMMost clients ignore these and don't even bother to add them to their routing table. They just send everything to the default router.
I guess by "most" you include mainstream operating systems like Windows, Mac OS, Linux, even FreeBSD? How can they claim to support IPv6 if they do not follow the relevant RFCs? WTH? That was quite a wake up call. I used to take all of this for granted and fully supported. Of course!
RFCs are the law. Period.
🤷
Quote from: Patrick M. Hausen on January 02, 2026, 02:01:36 AMI guess by "most" you include mainstream operating systems like Windows, Mac OS, Linux, even FreeBSD?
Yes. Didn't test every single one though, mostly typical client OSes.
Quote from: Patrick M. Hausen on January 02, 2026, 02:01:36 AMI used to take all of this for granted and fully supported.
So did I for a long time, because that's how it's supposed to work. Until I actually tried it. For example, I once wanted to set up a VPN gateway as a dedicated system, completely independent of the main firewall. Just plug it in the LAN and clients should start using it for accessing a remote site which uses a specific prefix. So it advertised itself as a router for this prefix. Clients simply ignored these RAs. Some say it's for "security reasons" to prevent traffic hijacking (I thought that's what filtering RAs on switches is for?).
Quote from: Patrick M. Hausen on January 02, 2026, 02:01:36 AMRFCs are the law. Period.
I guess we need an RFC court then...
Well... Crap. Before discarding this setup, here are my final questions:
- RFCs are being mentioned here multiple times, can someone call one specific out?
- By any chance, does anybody know which program is receiving the RAs under Linux? As the problem is present under Android and Debian, I suspect the kernel itself, which is somewhat odd... At least this would be a start for a proper fix.
- Similar to before: By any chance some ideas of how to add the src-addr to the Linux IPv6 route table for multipath route-entries? Which program manages these?
- I can't be the first one stumbling into this... Any ideas for keywords to look for? Maybe you also found some resource on that before? Anything to get a more clear view on this issue and potentially being able to get hold of the right persons...
Here is an excerpt for the IPv6 routing table (via "ip -6 route"), which suspiciously uses multipath routing, but these entries do not specify the proper source-address... But this is working fine for the other entries, so the selection is working well there?
2003:f5:7f19:954a::/64 dev enp5s0f0 proto ra metric 100 pref medium
2003:f5:7f19:958a::/64 dev enp5s0f0 proto ra metric 100 pref medium
2003:f5:7f2f:924a::2000 dev enp5s0f0 proto kernel metric 100 pref medium
2003:f5:7f2f:924a::/64 dev enp5s0f0 proto ra metric 100 pref medium
2003:f5:7f2f:928a::/64 dev enp5s0f0 proto ra metric 100 pref medium
fe80::/64 dev tun42 proto kernel metric 256 pref medium
fe80::/64 dev enp5s0f0 proto kernel metric 1024 pref medium
default proto ra metric 100 pref medium
nexthop via fe80::62be:b4ff:fe1f:7878 dev enp5s0f0 weight 1
nexthop via fe80::62be:b4ff:fe1d:f824 dev enp5s0f0 weight 1
Thank you very much for all your input! This at least reassures me that I'm somewhat not the idiot here :P
Start here:
SLAAC: https://www.rfc-editor.org/rfc/rfc4862
Multihoming: https://www.rfc-editor.org/rfc/rfc7157
FYI... I just decided to use a workaround of a more loose "Allow IPv6 Internet" firewall rule, which is now still bound to the LAN interface, but does not actually check the sources address anymore, like under IPv4. I know this is not perfect, but this allows the clients to randomly choose an IPv6 address, prefix and gateway and to more equally distribute across the OPNsense instances.
One should keep in mind, that this only works here, as both OPNsense IPv6 prefix delegations are originating from the same /56 IPv6-prefix. With different IPv6-prefixes, one would indeed need to create a locally administered prefix and perform e.g. NPT with RA-CARPs...
Interesting approach. This can cause asymmetric routing though (client sends packets to OPNsense #1, upstream router (Fritzbox) sends reply to OPNsense #2). Just keep this in mind when possibly troubleshooting issues in the future.
RFC 8028 covers this use case by the way ("First-Hop Router Selection by Hosts in a Multi-Prefix Network").
And a fairly recent talk about why this mostly doesn't work in the real world (yet): https://media.ccc.de/v/denog17-78841-ipv6-multihoming-without-bgp-quo-vadis
Absolutely, I'm also aware of this issue and was already somewhat surprised that it worked instantly... We will see if I encounter more serious issues down-the-road, but for now most of my IPv6 network runs at least.
Now I started running into issues with Docker and IPv6 (due to no NAT the source-address being masked), which are also just painful - a topic for another day...
Quote from: Patrick M. Hausen on January 02, 2026, 01:32:52 AMQuote from: Maurice on January 02, 2026, 01:27:15 AMThis is simply unsupported in the real world. You can't have multiple default routers which advertise different SLAAC prefixes in the same LAN.
I noticed that "in the real world" but still this comes as quite the surprise to me. Because all the IPng/IPv6 fundamentals textbooks I read more than a decade ago said that is how it's supposed to work. Multihoming - solved. An arbitrary number of addresses/prefixes on every host interface plus router advertisements will just do the right thing.
So like the OP I took it as a given that this setup should "just work".
But then again at some point in time host to host IPsec was mandatory for any compliant IPv6 implementation, too ...
Yes, I tread that path, but there's a slight problem in the RFCs.
IIRC routing works like this:
- client has destination ip
- client selects gateway for that ip
- there is more than one gateway with default route
- those gateways are identified by link-local addresses
- selection is pretty much random, since all are default and same metric and so on
- client selects a source address for that gateway
- this source address selection is not guaranteed to be the right one, since routing information is not bound to addresses
- client sends packet to the gateway with the selected source ip
- gateway forwards packet, if rules allow it
- if the source address was the right one for the gateway, the same gateway receives the return packet and everything is fine
- if the source address was the one for the other gateway's network, then the other gateway receives the return packet and drops it
Worse, even if that scheme worked (which it does if the gateways in question are stateless, which firewalls are not), the routing decision is left to the client, which can not always know if the chosen gateway is right now capable of forwarding packets to the destination. This decision should be left to the gateways themselves, which necessitates the CARP solution.
But even then, you need a static ipv6 network you can configure on both firewalls, which is probably not possible using the fritzbox.
There is no good solution for this, I'm afraid. The google keyword is "ipv6 small site multihoming". There is also a candidate RFC for all the ways in which it doesn't work.