Hi!
Long time lurker, first time poster. Let's get started by describing my setup, so the following noticed misbehavior makes more sense to you.
SetupUplink (Fritz!Box) -> OPNsense#1 -> LAN
\-> OPNsense#2 -> LAN
Both OPNsense instances are configured as HA via a dedicated RJ45 connection, and #1 synchronizes to #2. The Fritz!Box is obtaining an /56 IPv6 prefix from the ISP and has prefix delegation enabled. This allows both OPNsense instances to obtain an /58 IPv6 prefix for their WAN interfaces. Let's add some more concrete number to this:
// Fritz!Box
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9500::/56 (two /57 are then derived for home-network and guest-network)
// OPNsense#1
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9580::/58
MAC in LAN: 60:be:b4:1d:f8:24
IPv6 link-local in LAN: fe80::62be:b4ff:fe1d:f824/64
// OPNsense#2
Dynamic IPv6 prefix on WAN: 2003:f5:7f19:9540::/58
MAC in LAN: 60:be:b4:1f:78:78
IPv6 link-local in LAN: fe80::62be:b4ff:fe1f:7878/64
ClientI have at least one client connected to the LAN network. In reality, multiple different Linux-based clients do exist, like Android, Linux Mint or Debian - all experience the issue. After connecting, this is the IPv6-configuration (I removed IPv4 here for simplification) of one Linux-machine (Linux 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux):
4: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:90:fa:e5:b6:9a brd ff:ff:ff:ff:ff:ff
inet6 2003:f5:7f19:954a::2000/128 scope global dynamic noprefixroute
valid_lft 5346sec preferred_lft 2646sec
inet6 2003:f5:7f19:954a:4da2:82bb:4829:2cae/64 scope global temporary dynamic
valid_lft 86119sec preferred_lft 14119sec
inet6 2003:f5:7f19:954a:560e:ff4a:2078:76b0/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 86119sec preferred_lft 14119sec
inet6 2003:f5:7f19:958a:906e:97a2:c6e3:7560/64 scope global temporary dynamic
valid_lft 86159sec preferred_lft 14159sec
inet6 2003:f5:7f19:958a:7eba:6d14:9339:c9f4/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 86159sec preferred_lft 14159sec
inet6 fe80::6b62:2f6f:8e71:3de2/64 scope link noprefixroute
valid_lft forever preferred_lft forever
...so, we can see that the machine received the RAs from both OPNsense instances and has chosen two IPv6 addresses for each of them: One "temporary" and one "mngtmpaddr".
The IssueLet's ping some IPv6 from the affected system:
ping6 2606:4700:4700::1111...whoops - no response! Sometimes the command does actually work, sometimes not. Reconnecting or waiting does help, until it stops working once again...
Tracing the IssueOkay. Why? Let's take a look into the firewall logs from OPNsense#1:
__timestamp__ 2026-01-01T22:44:11
action [block]
anchorname
class 0x00
dir [in]
dst 2606:4700:4700::1111
hoplimit 64
interface vlan04
ipversion 6
label Custom reject / state violation rule
length 64
protoname ipv6-icmp
protonum 58
reason match
rid 86dbbe50f6310147e4cafeb1ed54d663
src 2003:f5:7f19:954a:4da2:82bb:4829:2cae
Hmm - the ping request got sent to the OPNsense#1 instance and got dropped there, as the "Allow IPv6 internet" rule did not match and got catched by my "Custom reject / state violation rule" (just logs and rejects any not matched package similar to the default block rule). Indeed, this is because of the src-address: 2003:f5:7f19:954a:4da2:82bb:4829:2cae,
which is matching the IPv6 prefix of OPNsense #2!So, the clients in question seem to randomly sent the traffic of their IPv6 source address from e.g. OPNsense#2 to the OPNsense#1 instance, which in turn drops the packets, as their do not originate from its interface IPv6 prefix.
Let's confirm, that the correct link-local addresses are part of the router-advertisements, so the clients know the correct MAC for the advertised IPv6-prefix. Filtering via "icmpv6.type == 134" in Wireshark upon connecting the client interface, two advertisements are received:
Ethernet II, Src: SBluetech_1d:f8:24 (60:be:b4:1d:f8:24), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1d:f824, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Prefix information : 2003:f5:7f19:958a::/64)
Ethernet II, Src: SBluetech_1f:78:78 (60:be:b4:1f:78:78), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1f:7878, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Prefix information : 2003:f5:7f19:954a::/64)
...so the prefix 2003:f5:7f19:958a::/64 should be associated with fe80::62be:b4ff:fe1f:7878. Let's check the IPv6 ICMP Neighbor Solicitation for that link-local address:
Ethernet II, Src: SBluetech_1f:78:78 (60:be:b4:1f:78:78), Dst: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a)
Internet Protocol Version 6, Src: fe80::62be:b4ff:fe1f:7878, Dst: fe80::6b62:2f6f:8e71:3de2
ICMPv6 Option (Source link-layer address : 60:be:b4:1f:78:78)
...so the link-local address fe80::62be:b4ff:fe1f:7878 is associated with 60:be:b4:1f:78:78.
Let's see if the ICMPv6 echo requests are also sent there:
Ethernet II, Src: Emulex_e5:b6:9a (00:90:fa:e5:b6:9a), Dst: SBluetech_1f:78:78 (60:be:b4:1f:78:78)
Internet Protocol Version 6, Src: 2003:f5:7f19:958a:ea3b:e9b1:abf7:aa03, Dst: 2606:4700:4700::1111
Huh?! The client used its address from the 2003:f5:7f19:958a::/64 IPv6 prefix from OPNsense#1 to sent to the 60:be:b4:1f:78:78 MAC of OPNsense#2.
What?!As far as I understand IPv6 networking, the client should have chosen the other destination MAC from OPNsense#1 for sending the package, so it would have arrived at the correct instance. The filtering rule(s) in OPNsense#1/#2 are correct, as their are restricting the source-addresses to originate from their advertised IPv6 prefixes and discard (wrong) addressed traffic before forwarding.
Some more thoughts?
- In these multi-gateway IPv6 networks, are all gateways required to serve all locally advertised IPv6 prefixes? No, this does not make sense, as they could differ and not every gateway would be able to forward traffic for all prefixes. Also, ICMPv6 Router Advertisements do specify the prefix length, hence the client can figure out which one to pick for its chosen address.
- Bug in Linux Kernel? Meh, IPv6 is old, that I doubt this is bug would have flown that long under the radar. Maybe something in higher layers? Like selection of binding sockets to addresses in TCP? But why would then also ICMP fail? Same src-addr selection logic used, but then really in the kernel?!
- Are there some RFCs requiring this behavior?
- Am I misunderstanding the resolution of MACs in IPv6? Are Route Advertisements and source-address selection really linked like this?
So yeah... I'm somewhat confused what I find here... Do you have any more ideas? Are there details for you to understand missing? Yes, I have not masked any IPv6-prefixes or MACs here intentionally, as maybe their specific values trigger this behavior?
Looking forward to some ideas :P
In an HA cluster you are supposed to create a CARP address on LAN, preferably link-local. I use for example fe80::<vlan-id> on all networks. Then use that address as the source for your router advertisements. Works rock solid.
@Patrick Hmm, this does not sound right as we are talking about IPv6 here and the two OPNsense instances are receiving two *distinct* IPv6 prefixes (due to design derived from same /56 prefix). Yes, for IPv4 I fully understand (and implemented) CARP addresses for the IPv4 gateway address.
Instead, with this setup, the clients should be able to use both IPv6 prefixes at their own discretion. Furthermore, if you use CARP for the RA-source address, then the client can only use the IPv6 prefix of the current CARP-MASTER, as the traffic for the other one would be once again dropped as the BACKUPs prefix would be unknown to the MASTER - right? Hence, defeating the idea of simultaneous use of both IPv6 prefixes.
Maybe you can share some reference regarding the IPv6 CARP address use for Router Advertisements? Especially since that recommendation would then still not explain the issue I'm observing and be only a "fix" by breaking one set of derived IPv6 addresses for the clients.
The concept of a HA cluster installation of OPNsense is to look like a single device to all clients for both IPv4 and IPv6. This is what works. What you are trying to build is something I have never done, is not in the docs, so you are on your own treading new paths here.
Possibly there is a bug. I understand now that you want to advertise two different prefixes with two different routers. But that is not "OPNsense HA" as devised by the HA implementation. OPNsense HA implies CARP and failover for everything. Regardless of the protocol.
Quote from: Patrick M. Hausen on January 01, 2026, 11:40:20 PMThe concept of a HA cluster installation of OPNsense is to look like a single device to all clients for both IPv4 and IPv6. This is what works. What you are trying to build is something I have never done, is not in the docs, so you are on your own treading new paths here.
Indeed, I want to use dynamic IPv6 prefixes, therefore I cannot hard-code both OPNsense instances to advertise the same IPv6 prefix, for which then a IPv6 CARP gateway would make sense.
Quote from: Patrick M. Hausen on January 01, 2026, 11:40:20 PMyou are on your own treading new paths here.
Possibly there is a bug.
From my understanding, OPNsense is not really doing anything wrong here... The HA-aspect is in reality even not applicable here, as it only synchronizes the config and the conntrack-states (...), which are not share-able with this setup.
Quote from: Patrick M. Hausen on January 01, 2026, 11:40:20 PMI understand now that you want to advertise two different prefixes with two different routers. But that is not "OPNsense HA" as devised by the HA implementation. OPNsense HA implies CARP and failover for everything. Regardless of the protocol.
Indeed. You are absolutely right. I have updated the post title accordingly... Any further ideas? I really want to avoid escalating this into the kernel mailing lists, as they are always being spammed too much by mailinglist-address-scrapers :P
This is simply unsupported in the real world. You can't have multiple default routers which advertise different SLAAC prefixes in the same LAN.
While hosts theoretically SHOULD use the router which matches the selected source address, pretty much no OS does this. It's not just a Linux issue.
Cheers
Maurice
Quote from: Maurice on Today at 01:27:15 AMThis is simply unsupported in the real world. You can't have multiple default routers which advertise different SLAAC prefixes in the same LAN.
I noticed that "in the real world" but still this comes as quite the surprise to me. Because all the IPng/IPv6 fundamentals textbooks I read more than a decade ago said that is how it's supposed to work. Multihoming - solved. An arbitrary number of addresses/prefixes on every host interface plus router advertisements will just do the right thing.
So like the OP I took it as a given that this setup should "just work".
But then again at some point in time host to host IPsec was mandatory for any compliant IPv6 implementation, too ...
Quote from: Patrick M. Hausen on Today at 01:32:52 AMBecause all the IPng/IPv6 fundamentals textbooks I read more than a decade ago said that is how it's supposed to work.
Indeed, but unfortunately it doesn't. And not just for the "multiple default routers" use case. Another example which doesn't work in the real world are routers which don't advertise themselves as a default router (Router Lifetime 0), but do advertise more specific routes. Most clients ignore these and don't even bother to add them to their routing table. They just send everything to the default router.
Quote from: Patrick M. Hausen on Today at 01:32:52 AMMultihoming - solved.
Multihoming is a beautiful thought experiment specified in many RFCs, plenty of which are ignored by real-world implementations...
Quote from: Maurice on Today at 01:54:15 AMMost clients ignore these and don't even bother to add them to their routing table. They just send everything to the default router.
I guess by "most" you include mainstream operating systems like Windows, Mac OS, Linux, even FreeBSD? How can they claim to support IPv6 if they do not follow the relevant RFCs? WTH? That was quite a wake up call. I used to take all of this for granted and fully supported. Of course!
RFCs are the law. Period.
🤷
Quote from: Patrick M. Hausen on Today at 02:01:36 AMI guess by "most" you include mainstream operating systems like Windows, Mac OS, Linux, even FreeBSD?
Yes. Didn't test every single one though, mostly typical client OSes.
Quote from: Patrick M. Hausen on Today at 02:01:36 AMI used to take all of this for granted and fully supported.
So did I for a long time, because that's how it's supposed to work. Until I actually tried it. For example, I once wanted to set up a VPN gateway as a dedicated system, completely independent of the main firewall. Just plug it in the LAN and clients should start using it for accessing a remote site which uses a specific prefix. So it advertised itself as a router for this prefix. Clients simply ignored these RAs. Some say it's for "security reasons" to prevent traffic hijacking (I thought that's what filtering RAs on switches is for?).
Quote from: Patrick M. Hausen on Today at 02:01:36 AMRFCs are the law. Period.
I guess we need an RFC court then...