OPNsense Forum

I spent almost the entire day trying to get IPv6 to work.

My network setup looks like this:

INTERNET <----> OpenWRT Router <---> OPNSense Router 1 (CARP Master on LAN)
                                                    <---> OPNSense Router 2 (CARP Slave on LAN)


TLDR: Now it finally started to work (IPv6 AND IPv4), although Firefox Fallback doesn't appear to be working.

At one point everything stopped working almost by itself. I couldn't even reach the OpenWRT Router from my LAN anymore using IPv4. It seemed that the whole Routing stopped working. I also tried to Manually Add Outbound NAT but to no avail.

The only things that appeared to have solved the Problem were:
- Delete the IPv4 Gateway for WAN and let OPNSense recreate it
- UNCHECK Dynamic gateway policy in Interfaces -> LAN ("This interface does not require an intermediate system to act as a gateway")

I guess IPv6 is the perfect example of "NAT is not Firewall" kind of thinking. My private (LAN) IP address is essentially my Public IP address, since the IPv6 Delegation Prefix has been requested by OPNSense.

Is that by design or did I do some misconfiguration on my end ?

My ISP gives me a /48 Subnet, so I let OpenWRT manage/allow Delegations "up to" /48, and each OPNSense is using "just" a /50. So theoretically, this should let me have 3 Routers on that switch with /50 each (because both OpenWRT and each of the OPNSense Routers is already consuming 1 IP).

There are however plenty of things that are "weird"/not optimal:

- The DHCP6 Server Subnet do NOT overlap because the WAN IP "Block" (the /50 Delegation Prefix) are 2 consecutive Blocks. So basically each Client in the LAN is getting 2 IPv6 Addresses (I'm getting like 3-4 actually + the link-local fe80:: ... I guess because the DHCP Servers reply at different speeds, so sometimes the Slave is faster). So I cannot configure CARP like this, since the Subnets don't overlap  :(.

- Is it normal that my Private IP is the same as my Public IP now ? It feels like IP are not tied to an Interface/Network anymore (I guess I miss NAT ;D)

- Even though I configured (maybe badly ???) Router Advertisement in "Assisted" Mode, all the Leases are listed in Services -> ISC DHCPv6 -> Leases. Is that normal ?

Thank you for your help  :).

Hi,

as a reminder:
NAT is not a feature. It's a neccessary workaround in IPv4. In IPv6 you're online with a dedicated address. Changing constantly when privacy ext. is enabled.

NAT does not provide security. Your firewall does.

DHCPv6 is not required unless you do prefix delegation. Using the flag assisted you specify that clients can basically chose between DHCP and SLAAC. Android choses SLAAC, windows choses DHCP. Hence your IP address will be listed on the leases tab.

A host with IPv6 connectivity usually has a link-local adress and 1-n global unique adresses. Privacy extensions do generate one every hour and keep the old one. With assisted you may have a SLAAC and a DHCP assigned adress. In a standard /64 network this can be neglected.

A transfer net can be set up without GUAs. When all hosts are routers link-local adresses are fine. However, to access internet (e.g. for updates) they would require one.

Hope that helps to come up with better expectations

So basically, it's working as it should ?

I just checked ... Actually my Android Phone stopped having Internet Access after this IPv6 Configuration. Weird ... At the very least it should fallback to IPv4 I would expect.

I just checked 2 GNU/Linux Hosts and they have no Problem.

But maybe it's because the WiFi System (UNIFI) doesn't have IPv6 configured on each Access Point and the Controller ?

Modern browsers use happy eyeballs to decide between v4 and v6 https://datatracker.ietf.org/doc/html/rfc8305

Unifi AP's don't need to support IPv6 management to carry IPv6 traffic. Recent AP's support multicast well enough for it to work.

Bart...

I just checked again on the Android Phone ... IPv4 seems to work again. IPv6 doesn't work at all.

But my goodness the UNIFI Controller ... getting IPv4 to work is as simple as just saying "Just query the [main] DHCP Server".

For IPv6 on the other hand  ???

Not sure how most people do it ... Do you make 1 VLAN for each "Service Group", and on that VLAN you assign a separate say /54 delegated DHCPv6 Prefix, similarly to what you would do in IPv4 by assigning e.g. 10.10.0.0/20 or 172.20.0.0/20 or 192.168.0.0/24 or similar ?

So basically "every service is part of the [ISP delegated] IPv6 prefix /48 INTERNET-wise, but then the individual subnets are identified locally by restricting/shrinking the subnet even more, say /56 LOCALLY" ?

Quote from: luckylinux on May 12, 2024, 10:41:39 PM
So basically "every service is part of the [ISP delegated] IPv6 prefix /48 INTERNET-wise, but then the individual subnets are identified locally by restricting/shrinking the subnet even more, say /56 LOCALLY" ?

I don't parse that question 100%, but for one general statement: in IPv6 every broadcast domain (i.e. "Ethernet like network) is a /64. Always.

Here in Germany customers of German Telekom get a /56, which means you can have 256 separate VLANs with one /64 each.

If you are in the lucky position to get a /48 from your ISP that means you can simply (not walk into Mordor) e.g. have 256 locations in your corporate networks with then 256 VLANs each.

You never assign a longer prefix than /64 to a single interface.

HTH,
Patrick

Here in Argentina we are not so lucky. We get a /64 network from our ISP and if we want VLANs we need to subnet it.
I manage to assign /80 to each of my VLANs and forget to have IPv6 for smart phones. :(

Quote from: luckylinux on May 12, 2024, 10:41:39 PM
But my goodness the UNIFI Controller ... getting IPv4 to work is as simple as just saying "Just query the [main] DHCP Server".

For IPv6 on the other hand  ???

This is not a unifi forum. Why don't you just buy a USG?

BTW: You assign /64 subnets for segments. You can however delegate more prefixes to more subnets reachable within that segment via routers.

Quote from: Patrick M. Hausen on May 12, 2024, 11:23:06 PM

Quote from: luckylinux on May 12, 2024, 10:41:39 PM
So basically "every service is part of the [ISP delegated] IPv6 prefix /48 INTERNET-wise, but then the individual subnets are identified locally by restricting/shrinking the subnet even more, say /56 LOCALLY" ?

I don't parse that question 100%, but for one general statement: in IPv6 every broadcast domain (i.e. "Ethernet like network) is a /64. Always.

Here in Germany customers of German Telekom get a /56, which means you can have 256 separate VLANs with one /64 each.

If you are in the lucky position to get a /48 from your ISP that means you can simply (not walk into Mordor) e.g. have 256 locations in your corporate networks with then 256 VLANs each.

You never assign a longer prefix than /64 to a single interface.

HTH,
Patrick

I read things about /64 Subnet Size with IPv6 and must say I'm quite Confused there as well.

I'm pretty sure I read /64 is the MINIMUM Subnet Size [for Delegation ?].

But then it's true that when my OPNSense Firewalls get their "own" WAN IP from OPNSense, then they are getting a /128 (single Host).

I also read some of your Posts on this Forum Patrick (not sure if directly related to my current Issue) about CARP and IPv6.

I am getting an IPv6 /48 Prefix

2aXX:XXXX:XXXX::/48 -> Delegation from my ISP

2aXX:XXXX:XXXX::1/48 -> What OpenWRT Router can manage/is managing on the LAN Side (Public IP)
fdXX:XXXX:XXXX::1/48 -> What OpenWRT Router can manage/is managing on the LAN Side (Local ULA)

2aXX:XXXX:XXXX:4000::/50 -> goes to OPNSenseRouter1 (Public IP)
fdXX:XXXX:XXXX:4000::/50 -> goes to OPNSenseRouter1 (Local ULA)

2aXX:XXXX:XXXX:8000::/50 -> goes to OPNSenseRouter2
fdXX:XXXX:XXXX:8000::/50 -> goes to OPNSenseRouter1 (Local ULA)

Here the first Problem / Non-Optimum: due to CARP, I'm "losing" half the available IP Subnets AND Duplicating all IPs (due to DHCP Server timing, each host might get 1 IP from OPNSenseRouter1 and 1 IP from OPNSenseRouter2).

Then I can split each /50 into (2^(64-50-1) = 8192 ?) Separate /64 Networks, which I might as well group into VLANs (sigh ... due to Mellanox ConnectX2/3 Limitations, I believe a Maximum of Only 125-128 VLANs is supported anyway).

And of course each of these ranges, I will most likely split into 2, similarly to what I do with IPv4 (lower part for statically assigned / reservation of IP Addresses, higher part for dynamically assigned IP Addresses).

But isn't there a way for the 2 OPNSense Routers to work together "on the same Dedication Prefix" say /49 (instead of 2 x /48 with all the duplication that comes with it) ?

I could only get OpenWRT to give out the requested Delegation Prefix in "Basic Mode".
In "Advanced Mode", either specifying the "ia-pd 1" or "ia-na 1" etc, does not work at all.

And I guess it's a feature, not a bug, that the 2 OPNSense instances cannot get the same Delegation Prefix from the upstream Router, or ?

If you really get a /48 from your ISP isn't that a static one? So configure all interfaces statically. CARP works in this case. I don't know if prefix delegation and a HA cluster play together well, but as a guess I doubt it.

Quote from: Patrick M. Hausen on May 13, 2024, 09:48:54 AM
If you really get a /48 from your ISP isn't that a static one? So configure all interfaces statically. CARP works in this case. I don't know if prefix delegation and a HA cluster play together well, but as a guess I doubt it.

To be honest I don't know if IPv6 Prefix is Static. I purchased a separate static IPv4 from my ISP, but I don't know about the "default" IPv6 Subnet.

I'd also guess that HA with Prefix Delegation could cause issue.

So on both OPNSense Routers, you propose for the WAN that I should select a "Static IPv6" and use the /49 Subnet on both, with IP maybe ::11 for Router1 and ::12 for Router2 ?

Then for LAN (and all other interfaces: DMZ, WiFi, VLAN_XXX), I setup as "Track Interface" as usual with id 0,1,2,3,.... ?

Quote from: luckylinux on May 13, 2024, 09:53:12 AM
So on both OPNSense Routers, you propose for the WAN that I should select a "Static IPv6" and use the /49 Subnet on both, with IP maybe ::11 for Router1 and ::12 for Router2 ?

Then for LAN (and all other interfaces: DMZ, WiFi, VLAN_XXX), I setup as "Track Interface" as usual with id 0,1,2,3,.... ?

No, no no ... for WAN there is obviously some transfer network in place so you ask your ISP what exactly to configure.

And you never configure anything but /64 (or for WAN in some cases /128) on a single interface.

Then you pick one /64 from your /48 for LAN and configure one address from that prefix on the master and another address from the same prefix on your slave. This is just like IPv4 - both nodes share a single network on each interface.

Quote from: Patrick M. Hausen on May 13, 2024, 10:15:38 AM

Quote from: luckylinux on May 13, 2024, 09:53:12 AM
So on both OPNSense Routers, you propose for the WAN that I should select a "Static IPv6" and use the /49 Subnet on both, with IP maybe ::11 for Router1 and ::12 for Router2 ?

Then for LAN (and all other interfaces: DMZ, WiFi, VLAN_XXX), I setup as "Track Interface" as usual with id 0,1,2,3,.... ?

No, no no ... for WAN there is obviously some transfer network in place so you ask your ISP what exactly to configure.

And you never configure anything but /64 (or for WAN in some cases /128) on a single interface.

Then you pick one /64 from your /48 for LAN and configure one address from that prefix on the master and another address from the same prefix on your slave. This is just like IPv4 - both nodes share a single network on each interface.

I'm giving them a call right now.

I understand your second part of the reply. Not so much the first one, sorry  ???.

>> No, no no ... for WAN there is obviously some transfer network in place so you ask your ISP what exactly to configure.

Am I NOT supposed to use a part of the Prefixed Subnet they assign me ? Because this is what I'm currently doing (and mentioned in the first post that my Private IP = Public IP, since no NAT).

EDIT 1: on the phone, the "IT Expert Consultant" of the ISP told me that he does NOT know how IPv6 works. He asked a colleague who told him that he never saw IPv6 Addresses change, but he cannot promise that it will never change. Nice to see when ISP IT Experts know even less about IPv6 than me as a Homelabber :D. "IPv6 is too complicated to setup I will never do it" he told me ???.

This was Technical Support, not even Customer Support  :D.

Frequently ISPs use a single IPv6 address for the external connection of their customers in addition to a delegated prefix.

For example I currently got 2003:a:d7f:d938:f690:eaff:fe00:ca67/64 on WAN while my delegated prefix is 2003:a:d59:3800::/56.

Like in IPv4 this is commonly called a transfer network, because it is not used for any services but simply so that the routers have dedicated addresses. Different from IPv4 it is not strictly necessary. Routing works fine over just link-local addresses.

Whatever is the case your ISP should be able to tell you but it seems they are not.  :o

From that /48 of yours you then pick an arbitrary /64 for LAN and assign e.g. ::1 to your first node and ::2 to the second. For CARP you should pick a link-local address that will then be announced to all your clients as the default gateway via router advertisments. E.g. fe80::1

Essentially IPv6 is way simpler than IPv4. Only forget you must what you have learned  :)

Try prefix delegation ("track interface) on both nodes and set the prefix hint to "0" on both. They should have a single common /64 on LAN - only then can you proceed with CARP etc.

Quote from: Patrick M. Hausen on May 13, 2024, 11:22:16 AM
Frequently ISPs use a single IPv6 address for the external connection of their customers in addition to a delegated prefix.

For example I currently got 2003:a:d7f:d938:f690:eaff:fe00:ca67/64 on WAN while my delegated prefix is 2003:a:d59:3800::/56.

Like in IPv4 this is commonly called a transfer network, because it is not used for any services but simply so that the routers have dedicated addresses. Different from IPv4 it is not strictly necessary. Routing works fine over just link-local addresses.

Whatever is the case your ISP should be able to tell you but it seems they are not.  :o

From that /48 of yours you then pick an arbitrary /64 for LAN and assign e.g. ::1 to your first node and ::2 to the second. For CARP you should pick a link-local address that will then be announced to all your clients as the default gateway via router advertisments. E.g. fe80::1

Essentially IPv6 is way simpler than IPv4. Only forget you must what you have learned  :)

Try prefix delegation ("track interface) on both nodes and set the prefix hint to "0" on both. They should have a single common /64 on LAN - only then can you proceed with CARP etc.

I am getting a /128 and a /64 for use on the OpenWRT Router itself plus the /48 Prefix for LAN Delegation:
LAN Interface: 2aXX:XXX4:XXXX::1/48
WAN Interface: 2aXX:XXX0:XXXX:....:12c8/128
WAN Interface: 2aXX:XXX0:XXXX:....:2a4/64

Note that the Prefix for Delegation is different than the Public IP of the OpenWRT Router on the WAN Interface.
One is 2aXX:XXX4:... (for Prefix Delegation), the other one is 2aXX:XXX0: ... (for use by the OpenWRT WAN Interface itself).

Downstream the OPNSenseRouter1 gets ::7 and OPNSenseRouter2 gets ::8.

I might need to change those numbers since now ::1 is already taken by OpenWRT (I could change that though) ...

The "Reason" (at the time) was that I once had OPNSenseRouter1 IPv4 on 192.168.1.1 and 192.168.1.3 - 192.168.1.4 - 192.168.1.5 (and maybe 192.168.1.6) were already taken for the DNS Servers.

So when I introduced CARP with a second OPNSense (OPNSenseRouter2) I needed 3 addresses and therefore I picked 192.168.1.1 for the Virtual/CARP/Common IP (since that's the Gateway it's statically configured on many Client Devices and it's better not to change that) and 192.168.1.7/192.168.1.8 for the Individual Routers.

The only Technical Information they provided is this (translated by Google Translate):

Thank you for the conversation

here with what we have on Ipv6, hope it can help a little

IPv6

     Certain DHCPv6 settings must be set up on your router.
         SLAAC (point to point link between your router and our GW router).
         DHCPv6 with prefix delegation (DHCPv6-PD) (used to retrieve a delegated /48 prefix that can be distributed on the lan side of your router).
         Stateful Address Assignment | IA-NA = 1 (on)
         Identity Association for Prefix Delegation | IA-PD = 1 (on)
         Non-temporary address | IAID for NA = 1 (on) instead of 0 (off) which, according to our experience, will be the default setting in most DHCP client setups.
         We recommend using the DUID-LL [DUID type 3] algorithm (Algorithm used on routers issued by us).
         We recommend setting a static DUID-LL value (if possible) to avoid the value changing between restarts of your CPE (router) device.
     We assign you a /48 via DHCPv6-PD (Prefix Delegation).
         The /48 route is inserted into our routing table by our DHCPv6 relay.
         Without DHCPv6-PD you will not be able to use your /48 prefix.
     Our GW routers are set up to send router advertisements every 360 seconds (6m).


DUID (DHCP Unique Identifier)



     DUID-LLT: The Link-Layer address of one of the device's network interfaces, concatenated with a timestamp [RFC 2131].
     DUID-EN: An Enterprise Number plus additional information specific to the enterprise [RFC 2131].
     DUID-LL: The Link-Layer address of one of the device's network interfaces [RFC 2131].
     DUID-UUID: Used in situations where there is a Universally Unique IDentifier (UUID) stored in a device's firmware settings [RFC 6355].
     DUID-V6ADDR: "This document defines a new DHCPv6 Unique Identifier (DUID) type that contains a single 128 bit IPv6 address. Makes it possible for devices to use suitably-derived unique IPv6 addresses to identify themselves to DHCPv6 servers"



So I setup, on the OpenWRT Interface Properties, "Client ID to send when requesting DHCP" according to DUID-LL as they Reccomend. Nothing changed when I hit refresh though (neigher IPv4 nor IPv6, I guess it makes sense since it's the same as the default).

Unfortunately OpenWRT is quite confusing with wan/wan6 Interface, whereby wan (supposedly IPv4) contains some IPv6 Settings and wan6 (supposedly IPv6) contains some IPv4 Settings.

OPNSense itself has quite some quircks ... When asking upstream for /50 Delegation, there is no way to see that it actually gets it. In Interfaces -> Overview I always get a /128 plus a /64 (no matter what I do) plus a weird fe80::XXXXXX/64 which I did NOT configure anywhere for the WAN Interface (is that the link-local maybe ? Upstream Gateway is fd34:XXXX:XXXX::1 (OpenWRT Local ULA).

The only way I could see if OPNSense actually gets the /50 prefix is by looking at the OpenWRT DHCPv6 Leases page.

OK, so on WAN you configure the same prefix that the OpenWRT router has got on its LAN. Sorry, I missed the part that there is a full featured router in place already.

Just pick two addresses you like from that /64 and configure statically. Then as CARP configure a third address from that /64.

On the OpenWRT configure this third address as the gateway for the entire /48.

Then pick another /64 from your /48 and configure two static addresses with /64 prefix length on LAN of both your firewalls. For CARP configure e.g. fe80::<some small number you like, VLAN ID if existent, or simply "1">.

Enable router advertisements, unmanaged, on LAN.

Done.

I repeat it again:

Subnet is exactly /64
Subnets reachable from this subnet, get more /64 prefixes - obtained by prefix delegation (DHCPv6) or static configuration.

There is no segment with /48. You get your /48 from the ISP to assign /64 chunks to your segments. Not minimum, not maximum. Exactly!

Quote from: Saarbremer on May 13, 2024, 11:58:04 AM
There is no segment with /48. You get your /48 from the ISP to assign /64 chunks to your segments. Not minimum, not maximum. Exactly!

You are getting a /48 simply for address management purposes. IPv6 address space is that vast, we can afford that. You have two octets (bytes) worth of "structure".

Picture e.g. a corporate network with a handful of offices all connected via VPN to the HQ and policy is that all traffic to the Internet goes through the VPN tunnel and then out through the single central firewall cluster. Not uncommon.

That's why you get a /48. So you can give each location a /56. Octet boundaries are much more convenient to work with than, say, /51s or some such. You can have up to 256 locations that way, no small company has got that many, but so what? We have the address space.

And then at each location you can have up to 256 /64s for individual VLANs or otherwise separated networks. Again, you probably have 5 or 10 ... but who cares? 256 makes sure you never run out of address space.

As @Saarbremer wrote: each single network is exactly one /64. Always.

I run an ISP. Our assignment by RIPE is a /32. That means I have 2^32 networks of size /64 to work with. Yes, as many networks as the entire legacy (IPv4) Internet has got addresses! And each of these networks can theoretically hold a number of hosts that is the size of the legacy Internet squared! Although with Ethernet technology that isn't practical because you don't want broadcast domains larger than a couple of hundred hosts.

The /64 size was agreed upon so autoconfiguration is simple.

Quote from: Patrick M. Hausen on May 13, 2024, 11:56:58 AM
OK, so on WAN you configure the same prefix that the OpenWRT router has got on its LAN. Sorry, I missed the part that there is a full featured router in place already.

Just pick two addresses you like from that /64 and configure statically. Then as CARP configure a third address from that /64.

On the OpenWRT configure this third address as the gateway for the entire /48.

Then pick another /64 from your /48 and configure two static addresses with /64 prefix length on LAN of both your firewalls. For CARP configure e.g. fe80::<some small number you like, VLAN ID if existent, or simply "1">.

Enable router advertisements, unmanaged, on LAN.

Done.

>> Just pick two addresses you like from that /64 and configure statically. Then as CARP configure a third address from that /64.

On the OpenWRT LAN-side (OPNSense WAN-side) I currently have 2aXX:XXX4:XXXX::1/48.

Is that a mistake then ?

But shouldn't the OpenWRT LAN-side Bridge (OPNSense WAN-side) be the default gateway for ALL Addresses in the Range 2aXX:XXX4:XXXX::/48 ?

Maybe that's where I'm getting confused (for access to INTERNET, NOT LAN/DMZ/WiFi/etc): How can these "secondary" /64 Networks access the Internet Otherwise, if the Upstream Gateway (OpenWRT LAN-side) is on a different /64 Subnet ?

I'm pretty sure that would trigger the OPNSense Firewall "Default Deny State Violation" Error (since it's on a different Subnet).

Or would this be "solved" via Router Advertisement by pushing a "Default Route" to the upstream Gateway for the whole /48 Network ?

The OpenWRT router should have a /64 on its LAN side just like every other system

All interfaces are /64. Putting a /48 prefix length on an interface is wrong. Period. If that system was configured by your ISP they don't know IPv6.

Do you have administrative access to that router? Change the prefix length to /64, then proceed as I wrote above.

The OpenWRT is the default gateway for all systems that are directly connected to its LAN side. For all other /64s the default gateway is the OPNsense CARP address in that network.

Quote from: Patrick M. Hausen on May 13, 2024, 01:53:05 PM
The OpenWRT router should have a /64 on its LAN side just like every other system

All interfaces are /64. Putting a /48 prefix length on an interface is wrong. Period. If that system was configured by your ISP they don't know IPv6.

Do you have administrative access to that router? Change the prefix length to /64, then proceed as I wrote above.

The OpenWRT is the default gateway for all systems that are directly connected to its LAN side. For all other /64s the default gateway is the OPNsense CARP address in that network.

So basically it's going to be the OPNSense that's going to route from all the different /64 Networks to the Subnet of the OpenWRT Router ?

Isn't that either doing NAT or generating a "Default Deny State Violation" ?

EDIT 1: Yes, I can confirm. As soon as I "squeezed" the Network Interfaces Subnet to /64 (instead of /48), the end-Clients (Ubuntu GNU/Linux used for Testing) are getting a "Default deny / state violation rule" since the Subnet is different and OPNSense refuses to Route.

EDIT 2: Now on Services -> DHCPv6 I am getting "No available address range for configured interface subnet size." so end-Clients cannot get a DHCPv6 Lease any longer  :(.

Quote from: luckylinux on May 13, 2024, 02:01:53 PM
So basically it's going to be the OPNSense that's going to route from all the different /64 Networks to the Subnet of the OpenWRT Router ?

Yes.

Quote from: luckylinux on May 13, 2024, 02:01:53 PM
Isn't that either doing NAT or generating a "Default Deny State Violation" ?

With proper rules - no.

Quote from: Patrick M. Hausen on May 13, 2024, 02:41:01 PM

Quote from: luckylinux on May 13, 2024, 02:01:53 PM
Isn't that either doing NAT or generating a "Default Deny State Violation" ?

With proper rules - no.

Since it seems that IPv6 work on the OPNSense Routers (not that I'm sure I got the part about  the "Static Route" for CARP FROM OpenWRT TO OPNSense ... I guess this is in order to enable port forwarding ? - Not there yet), then I take it as I must MANUALLY configure some Routes in the Firewall -> Rules Section ?

I'm always confused because sometimes "System -> Static Routes" has to be used, sometimes "System -> Gateways" can "solve" the problem (or at least it did for the /48 Network Size, which is probably a mistake), sometimes it's Firewall -> Rules.

EDIT 1: lost all Internet Connectivity. Not sure where systemd-networkd on my Ubuntu GNU/Linux is getting the IPv6 Address from, but it's not in the list of DHCPv6 Leases. I also deleted the .leases files in /var/lib/... Weird

EDIT 2: I'm having a hard time understanding what is going on right now ... I enabled logging for ALL Rules. Ping OPNSense -> INTERNET Website works OK. Ping Clients -> OPNSense works OK. Ping Clients -> OpenWRT / INTERNET doesn't work.

But at the same time EVERYTHING is Green in the Firewall Logs for ICMP ... (at least for the Public IP Address).

Is there a requirement to add duplicated Rules for Local-Link Interfaces for Instance ? There is absolutely nothing red for ICMP Ping though in the Logs ...

EDIT 3: Issue seems to be when I setup CARP Outbound NAT for IPv4 -> IPv6 didn't have any. So I guess all traffic got silently dropped by OpenWRT ? Not sure (I tried to "Allow Invalid Traffic" etc, but that didn't help). As soon as I placed a "Outbound NAT" rule on OPNSense to match the CARP WAN IPv6 Address, then it started working.

But of course this means no IPv6 Public Address ...

EDIT 4: even with Outbound NAT for IPv6 it's EXCRUCIATING SLOW. Like 60 seconds before replying to Pings...

Right now everything is configured as Static IPv4/IPv6 for OPNSense (NOT the end-clients, i.e. GNU/Linux etc) in order to try to locate the Issue.

Quoteeven with Outbound NAT for IPv6 it's

<sound of me eating my desk>  ???

Quote from: Saarbremer on May 13, 2024, 06:14:02 PM

Quoteeven with Outbound NAT for IPv6 it's

<sound of me eating my desk>  ???

Don't tell me ... I'm completely lost.

OPNSense logs don't show anything. OpenWRT logs don't show anything.

tcptraceroute6 2001:4860:4860::8888 (Google DNS Server IPv6 #1) works correctly although a bit slow
traceroute6 2001:4860:4860::8888 (Google DNS Server IPv6 #1) times out after Reaching OPNSense Router
ping -6 2001:4860:4860::8888 doesn't work at all.

It seems to be more of a "glitch" when it works, rather than the opposite.

Either NAT and/or Routing is completely broken.

EDIT 1: the Client is getting these Addresses via DHCP, but ONLY the /128 is being used (I can see it in OPNSense Logs).

eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet6 2aXX:XXX4:XXXX:1:XXXX:XXXX:a9f2:1858/128 scope global deprecated
       valid_lft forever preferred_lft 0sec

    inet6 2aXX:XXX4:XXXX:1:XXXX:XXXX:fe79:2d9a/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86388sec preferred_lft 14388sec


EDIT 2: the other thing I could imagine is that the Packet gets correctly "out" of OPNSense towards the OpenWRT Router, then either:
- Gets discarded/rejected (bad Source Address ?) - Nothing in the logs though
- OpenWRT doesn't know how to "give" the response back

Sticky Connections are enabled in OPNSense so that should take care of it (on OPNSense side). I also tried "Bind States to Interfaces" but it didn't make a difference.

Very different behavior between ICMP and TCPTRACEROUTE though ... TCP Seems to work correctly and now Google DNS Server is very quick to reply (using tcptraceroute6).

Which is weird considered all the Allow / Pass rules for ICMP ...

EDIT 3: Added a Static Route from OpenWRT -> OPNSense CARP IP as Gateway (I guess this is needed so that OpenWRT knows where to send the ICMP Packets back, I guess this is what Patrick tried to explain before).

So now Ping & Traceroute work. DNS Stopped working. I don't know why it's always 1 step forward 1 step backward all the time ...

EDIT 4: It seems to work now ... Both tcptraceroute6, traceroute6 -I (ICMP) / -S (TCP) / -U (UDP) work correctly, so does PING. The missing link was the Static Route from OpenWRT back to the LAN which is controlled by OPNSense.

I guess this is/was done automatically in the case of IPv4 with NAT+Sticky connections, so when OPNSense receives the packet back, it knows where to forward it to within the LAN.

Of course now that IPv6 Addresses are "direct" and the LAN is not managed by OpenWRT, there is no way for OpenWRT to know where to send the Packet to, since he doesn't know the Hosts. Thus adding the Static Route is probably what fixed it.

I still don't know however why OpenWRT Lost IPv6 all of a sudden and the only solution to get it back was to reboot the OpenWRT Device ...

EDIT 5: And Android also works with IPv6 now (although using SLAAC). Kinda annoying that you cannot see what IP Address those devices get ... Other than maybe running periodic ARP / Network Scans ???

OK ... I REALLY hope somebody could bring some clarity to the current status.

• OpenWRT Fiber Router is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.
• OPNSense Router1 is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.
• OPNSense Router2 is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.

On the other Hand, OPNsense Router DHCP Clients, lose IPv6 Ping "capability" after a while.

Nothing shows up in Firewall -> Log Files -> Live View (well, what shows is "Green"  :D).

I'm suspecting an issue where the OPNsense CARP is misconfigured or misbehaving, like Packets are maybe going from LAN Client to OPNSense Router1, go to OpenWRT, then to the Internet, the reply comes back to OpenWRT, forwards to OPNSense CARP Adress and for some reason it goes to OPNSense Router2.

I'm not sure what to think otherwise ???.

No problem with IPv4, but IPv6 seems to last maximum a day, or as little as a few hours (didn't measure precisely ).

I wanted to use a Podman/Docker Container to monitor a Remote Host using IPv6 for this exact Reason, EXCEPT that Podman/Docker Rootless Containers don't really work well with IPv6 and external Routes to the Host.

So I cannot even monitor another HOST in the LAN with IPv6 at the moment from inside a Docker/Podman Container  :(.

Quote from: luckylinux on May 17, 2024, 10:23:35 AM
OK ... I REALLY hope somebody could bring some clarity to the current status.

• OpenWRT Fiber Router is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.
• OPNSense Router1 is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.
• OPNSense Router2 is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.

On the other Hand, OPNsense Router DHCP Clients, lose IPv6 Ping "capability" after a while.

Nothing shows up in Firewall -> Log Files -> Live View (well, what shows is "Green"  :D).

I'm suspecting an issue where the OPNsense CARP is misconfigured or misbehaving, like Packets are maybe going from LAN Client to OPNSense Router1, go to OpenWRT, then to the Internet, the reply comes back to OpenWRT, forwards to OPNSense CARP Adress and for some reason it goes to OPNSense Router2.

I'm not sure what to think otherwise ???.

No problem with IPv4, but IPv6 seems to last maximum a day, or as little as a few hours (didn't measure precisely ).

I wanted to use a Podman/Docker Container to monitor a Remote Host using IPv6 for this exact Reason, EXCEPT that Podman/Docker Rootless Containers don't really work well with IPv6 and external Routes to the Host.

So I cannot even monitor another HOST in the LAN with IPv6 at the moment from inside a Docker/Podman Container  :(.

You might want to debug packet flow using tcpdump on both opnsense firewalls.

Also, please post a complete network diagram with all interfaces and IPs, I can't really make sense of your setup.

Fair enough with the Network Diagram ... It's going to take a while -> DONE (see next post)

I also posted on OpenWRT Forum since I think it's more to do with some State Table / Firewall / Overload on the IPv6 Stack on their side:
https://forum.openwrt.org/t/ipv6-disappearing-after-a-while-for-lan-clients-troubleshooting-impossible-due-to-lack-of-openwrt-logs/198213

TLDR: once this Issue Occurs, restarting the WAN and WAN6 Interface on OpenWRT seems to solve the issue. Nothing else seemed to help (restarting LAN Client, restarting OPNSense Router1/Router2, restarting LAN Clients, etc).

Once that is done **on OpenWRT Router**, then the LAN Clients can once again Ping e.g. 2001:4860:4860::8888 (Google DNS Servers).

Attached Network Diagram.

For a bit added clarity:
- RED: segment of the IPv6 Delegation Prefix for WAN-interfaced Zones
- BLUE: segment of the IPv6 Delegation Prefix for LAN-interfaced Zones

Quote from: luckylinux on May 17, 2024, 11:22:06 AM
Attached Network Diagram.

For a bit added clarity:
- RED: segment of the IPv6 Delegation Prefix for WAN-interfaced Zones
- BLUE: segment of the IPv6 Delegation Prefix for LAN-interfaced Zones

This looks fine to me, provided the LAN clients get the CARP address as their gateway.

One thing perhaps, usually the gateway on LAN is a link-local address, so it should be preferred to use something like fe80::1 as CARP address on the LAN side and use that as gateway for the clients.
Clients like to talk to their routers using link-local addresses only.
The GUA CARP IP on LAN should then be unnecessary. It might well be that clients use the link-local addresses of the opnsense firewalls to route and then use the wrong one as their nexthop.

You should be able to find all of that out using tcpdump.

Quote from: bimbar on May 17, 2024, 11:56:50 AM

Quote from: luckylinux on May 17, 2024, 11:22:06 AM
Attached Network Diagram.

For a bit added clarity:
- RED: segment of the IPv6 Delegation Prefix for WAN-interfaced Zones
- BLUE: segment of the IPv6 Delegation Prefix for LAN-interfaced Zones

This looks fine to me, provided the LAN clients get the CARP address as their gateway.

One thing perhaps, usually the gateway on LAN is a link-local address, so it should be preferred to use something like fe80::1 as CARP address on the LAN side and use that as gateway for the clients.
Clients like to talk to their routers using link-local addresses only.
The GUA CARP IP on LAN should then be unnecessary. It might well be that clients use the link-local addresses of the opnsense firewalls to route and then use the wrong one as their nexthop.

You should be able to find all of that out using tcpdump.

But I thought the "Purpose" of IPv6 was to avoid NAT so end-devices could directly use IPv6 "Public" Addresses.

And use the IPv6 "Privacy Extension" (temporary Addresses) for Outbound Traffic, so that they don't get Tracked / Tagged by Websites, ISP, etc.

Otherwise why bother at all with IPv6 ? Are you "mapping" the Local-Remote Network in OPNSense using NPTv6 instead ?

You misunderstand, the clients use their public address as source, but the nexthop used is always link-local. Link-local is mainly for communication between clients and routers.

Which well fits the subject :D .

Quote from: bimbar on May 17, 2024, 12:32:53 PM
You misunderstand, the clients use their public address as source, but the nexthop used is always link-local. Link-local is mainly for communication between clients and routers.

Which well fits the subject :D .

>> One thing perhaps, usually the gateway on LAN is a link-local address, so it should be preferred to use something like fe80::1 as CARP address on the LAN side and use that as gateway for the clients.

So your comment was only for the CARP adress on the LAN side ? NOT Generally. Alright  :)

One thing that confuses me is that if I set fe80::1 for the LAN as CARP address, what am I going to use for the other Networks/VLANs in the future ? fe81::1 , fe82::1 , etc ?

Or rather use fe80:0001::1 for LAN, fe80:0002::1 for NET2, fe80:0003::1 for NET3, etc ?

Quote from: luckylinux on May 17, 2024, 12:36:44 PM

Quote from: bimbar on May 17, 2024, 12:32:53 PM
You misunderstand, the clients use their public address as source, but the nexthop used is always link-local. Link-local is mainly for communication between clients and routers.

Which well fits the subject :D .

>> One thing perhaps, usually the gateway on LAN is a link-local address, so it should be preferred to use something like fe80::1 as CARP address on the LAN side and use that as gateway for the clients.

So your comment was only for the CARP adress on the LAN side ? NOT Generally. Alright  :)

One thing that confuses me is that if I set fe80::1 for the LAN as CARP address, what am I going to use for the other Networks/VLANs in the future ? fe81::1 , fe82::1 , etc ?

Or rather use fe80:0001::1 for LAN, fe80:0002::1 for NET2, fe80:0003::1 for NET3, etc ?

Best case you would use fe80::1 for all networks.

Either fe80::1 for all. Link local addresses are local to a single link (surprise!) so using the same address on all interfaces is fine.

Quote from: Patrick M. Hausen on May 17, 2024, 01:06:00 PM
Either fe80::1 for all. Link local addresses are local to a single link (surprise!) so using the same address on all interfaces is fine.

Ah, cool, I didn't know that multiple Interfaces could all be identified as "fe80::1" and the System would still work ...

But where are you suggesting to put this for the CARP Interface ? On the OPNSense CARP Tutorial it says to use the CARP Address as "Source Adress" in the Router Advertisement Section: https://docs.opnsense.org/manual/how-tos/carp.html#setup-router-advertisments

So far I have it set to "Automatic" and no Route is Advertised (I think it's more reliable like this than to advertise the route for the Subnet etc).

Quote from: luckylinux on May 17, 2024, 01:59:13 PM

Quote from: Patrick M. Hausen on May 17, 2024, 01:06:00 PM
Either fe80::1 for all. Link local addresses are local to a single link (surprise!) so using the same address on all interfaces is fine.

Ah, cool, I didn't know that multiple Interfaces could all be identified as "fe80::1" and the System would still work ...

But where are you suggesting to put this for the CARP Interface ? On the OPNSense CARP Tutorial it says to use the CARP Address as "Source Adress" in the Router Advertisement Section: https://docs.opnsense.org/manual/how-tos/carp.html#setup-router-advertisments

So far I have it set to "Automatic" and no Route is Advertised (I think it's more reliable like this than to advertise the route for the Subnet etc).

Usually loopback addresses are addressed by "<IPv6 address>%<interface name>", so that it works with duplicate addresses and prefixes.
By default, if you manage addresses by SLAAC, then the source address of the router advertisement is automatically the default gateway. So you should do that as the howto specifies.

Just an Update from my side (I haven't touched OPNSense for now) ....

The Issue about LAN clients losing IPv6 Connectivity after 30 Minutes / 1 Hour / etc can be "worked around" by restarting (via CRON) the WAN/WAN6 Interface on OpenWRT Router. That's really a "hack" as it will also cause loss of IPv4 Connectivity for approx. 30 seconds over 30 Minutes (1800 seconds) so approx 2% of the Time.

This would be the Hack. BOTH WAN6 and WAN Interface need to be Restarted in order for this to Work.
**ESSENTIAL** is that WAN6 Interface **MUST** be reset **BEFORE** WAN6 Interface !

OpenWRT -> System -> Scheduled Tasks
(disabled since the better Solution has been implemented, see further down the Post, and that appears to work well ... at least for now)
# !! WAN6 Interface must be reset BEFORE WAN Interface !!
#1 * * * * /usr/local/bin/restart-wan-interfaces
#0 * * * * /usr/local/bin/restart-wan6-interfaces


WAN Interface Reset Script in /usr/local/bin/restart-wan-interfaces
#!/bin/sh

# Define Interface
interface="wan"

# Echo
echo "Bringing down Interface ${interface}"

# Stop Interfaces
ifdown ${interface}

# Wait a bit
sleep 5

# Echo
echo "Bringing up again Interface ${interface}"

# Start Interfaces
ifup ${interface}

WAN6 Interface Reset Script in /usr/local/bin/restart-wan6-interfaces
#!/bin/sh

# Define Interface
interface="wan6"

# Echo
echo "Bringing down Interface ${interface}"

# Stop Interfaces
ifdown ${interface}

# Wait a bit
sleep 5

# Echo
echo "Bringing up again Interface ${interface}"

# Start Interfaces
ifup ${interface}

Not Optimal, thus I researched a bit the OpenWRT Forums for Possible Clues ...

Primary Clue: https://forum.openwrt.org/t/loosing-ipv6-upstream-after-30-minutes/21131/21
Maybe also this: https://forum.openwrt.org/t/ipv6-works-only-with-wan-in-promiscuous-mode/490/18

So I applied the following Settings:

• OpenWRT -> Network -> Interfaces -> LAN Interface
  
  
  • Min RA interval: [3 seconds]
  • Max RA interval: [600 seconds] (same as default)
  • RA Lifetime: [1800 seconds] (same as default)
• OpenWRT -> Network -> Devices -> br-lan
  
  
  • Enable multicast querier [checked]
  • Enable multicast support: [checked]
  • Enable Promiscous Mode: [disabled] (apparently only one of multicast or promiscous mode is required, thus I disabled Promiscous Mode, since Multicast is "safer" and with less Traffic)
  • Force IGMP version: [no enforcement]
  • Force MLD version: [no enforcement]


• OpenWRT -> Network -> Devices -> wan
  
  
  • Enable multicast support: [checked]
  • Enable Promiscous Mode: [disabled] (apparently only one of multicast or promiscous mode is required, thus I disabled Promiscous Mode, since Multicast is "safer" and with less Traffic)
  • Force IGMP version: [no enforcement]
  • Force MLD version: [no enforcement]

Going strong after more than 24h with IPv6 on LAN clients working ... Knock on Wood.

It seems that old versions of OpenWRT also needed to have the "Allow-DHCPv6" Firewall Rule modified and REMOVE the SRC_IP and DEST_IP (see the explanation https://forum.openwrt.org/t/ipv6-works-only-with-wan-in-promiscuous-mode/490/17). That didn't seem necessary in the latest Stable Build though (23.05).

Overall it seems that the main Things that made this work is, in essence:

• Enable Multicast
• Use shorter time/period between RA Renewals (Min RA interval especially, since Max RA interval & Max RA interval are set at their default Values)