Confused about IPv6

[Patrick M. Hausen]

OK, so on WAN you configure the same prefix that the OpenWRT router has got on its LAN. Sorry, I missed the part that there is a full featured router in place already.

Just pick two addresses you like from that /64 and configure statically. Then as CARP configure a third address from that /64.

On the OpenWRT configure this third address as the gateway for the entire /48.

Then pick another /64 from your /48 and configure two static addresses with /64 prefix length on LAN of both your firewalls. For CARP configure e.g. fe80::<some small number you like, VLAN ID if existent, or simply "1">.

Enable router advertisements, unmanaged, on LAN.

Done.

[Saarbremer]

I repeat it again:

Subnet is exactly /64
Subnets reachable from this subnet, get more /64 prefixes - obtained by prefix delegation (DHCPv6) or static configuration.

There is no segment with /48. You get your /48 from the ISP to assign /64 chunks to your segments. Not minimum, not maximum. Exactly!

[Patrick M. Hausen]

There is no segment with /48. You get your /48 from the ISP to assign /64 chunks to your segments. Not minimum, not maximum. Exactly!

You are getting a /48 simply for address management purposes. IPv6 address space is that vast, we can afford that. You have two octets (bytes) worth of "structure".

Picture e.g. a corporate network with a handful of offices all connected via VPN to the HQ and policy is that all traffic to the Internet goes through the VPN tunnel and then out through the single central firewall cluster. Not uncommon.

That's why you get a /48. So you can give each location a /56. Octet boundaries are much more convenient to work with than, say, /51s or some such. You can have up to 256 locations that way, no small company has got that many, but so what? We have the address space.

And then at each location you can have up to 256 /64s for individual VLANs or otherwise separated networks. Again, you probably have 5 or 10 ... but who cares? 256 makes sure you never run out of address space.

As @Saarbremer wrote: each single network is exactly one /64. Always.

I run an ISP. Our assignment by RIPE is a /32. That means I have 2^32 networks of size /64 to work with. Yes, as many networks as the entire legacy (IPv4) Internet has got addresses! And each of these networks can theoretically hold a number of hosts that is the size of the legacy Internet squared! Although with Ethernet technology that isn't practical because you don't want broadcast domains larger than a couple of hundred hosts.

The /64 size was agreed upon so autoconfiguration is simple.

[luckylinux]

OK, so on WAN you configure the same prefix that the OpenWRT router has got on its LAN. Sorry, I missed the part that there is a full featured router in place already.

Just pick two addresses you like from that /64 and configure statically. Then as CARP configure a third address from that /64.

On the OpenWRT configure this third address as the gateway for the entire /48.

Then pick another /64 from your /48 and configure two static addresses with /64 prefix length on LAN of both your firewalls. For CARP configure e.g. fe80::<some small number you like, VLAN ID if existent, or simply "1">.

Enable router advertisements, unmanaged, on LAN.

Done.

>> Just pick two addresses you like from that /64 and configure statically. Then as CARP configure a third address from that /64.

On the OpenWRT LAN-side (OPNSense WAN-side) I currently have 2aXX:XXX4:XXXX::1/48.

Is that a mistake then ?

But shouldn't the OpenWRT LAN-side Bridge (OPNSense WAN-side) be the default gateway for ALL Addresses in the Range 2aXX:XXX4:XXXX::/48 ?

Maybe that's where I'm getting confused (for access to INTERNET, NOT LAN/DMZ/WiFi/etc): How can these "secondary" /64 Networks access the Internet Otherwise, if the Upstream Gateway (OpenWRT LAN-side) is on a different /64 Subnet ?

I'm pretty sure that would trigger the OPNSense Firewall "Default Deny State Violation" Error (since it's on a different Subnet).

Or would this be "solved" via Router Advertisement by pushing a "Default Route" to the upstream Gateway for the whole /48 Network ?

[Patrick M. Hausen]

The OpenWRT router should have a /64 on its LAN side just like every other system

All interfaces are /64. Putting a /48 prefix length on an interface is wrong. Period. If that system was configured by your ISP they don't know IPv6.

Do you have administrative access to that router? Change the prefix length to /64, then proceed as I wrote above.

The OpenWRT is the default gateway for all systems that are directly connected to its LAN side. For all other /64s the default gateway is the OPNsense CARP address in that network.

[luckylinux]

The OpenWRT router should have a /64 on its LAN side just like every other system

All interfaces are /64. Putting a /48 prefix length on an interface is wrong. Period. If that system was configured by your ISP they don't know IPv6.

Do you have administrative access to that router? Change the prefix length to /64, then proceed as I wrote above.

The OpenWRT is the default gateway for all systems that are directly connected to its LAN side. For all other /64s the default gateway is the OPNsense CARP address in that network.

So basically it's going to be the OPNSense that's going to route from all the different /64 Networks to the Subnet of the OpenWRT Router ?

Isn't that either doing NAT or generating a "Default Deny State Violation" ?

EDIT 1: Yes, I can confirm. As soon as I "squeezed" the Network Interfaces Subnet to /64 (instead of /48), the end-Clients (Ubuntu GNU/Linux used for Testing) are getting a "Default deny / state violation rule" since the Subnet is different and OPNSense refuses to Route.

EDIT 2: Now on Services -> DHCPv6 I am getting "No available address range for configured interface subnet size." so end-Clients cannot get a DHCPv6 Lease any longer  .

[Patrick M. Hausen]

So basically it's going to be the OPNSense that's going to route from all the different /64 Networks to the Subnet of the OpenWRT Router ?

Yes.

Isn't that either doing NAT or generating a "Default Deny State Violation" ?

With proper rules - no.

[luckylinux]

Isn't that either doing NAT or generating a "Default Deny State Violation" ?

With proper rules - no.

Since it seems that IPv6 work on the OPNSense Routers (not that I'm sure I got the part about  the "Static Route" for CARP FROM OpenWRT TO OPNSense ... I guess this is in order to enable port forwarding ? - Not there yet), then I take it as I must MANUALLY configure some Routes in the Firewall -> Rules Section ?

I'm always confused because sometimes "System -> Static Routes" has to be used, sometimes "System -> Gateways" can "solve" the problem (or at least it did for the /48 Network Size, which is probably a mistake), sometimes it's Firewall -> Rules.

EDIT 1: lost all Internet Connectivity. Not sure where systemd-networkd on my Ubuntu GNU/Linux is getting the IPv6 Address from, but it's not in the list of DHCPv6 Leases. I also deleted the .leases files in /var/lib/... Weird

EDIT 2: I'm having a hard time understanding what is going on right now ... I enabled logging for ALL Rules. Ping OPNSense -> INTERNET Website works OK. Ping Clients -> OPNSense works OK. Ping Clients -> OpenWRT / INTERNET doesn't work.

But at the same time EVERYTHING is Green in the Firewall Logs for ICMP ... (at least for the Public IP Address).

Is there a requirement to add duplicated Rules for Local-Link Interfaces for Instance ? There is absolutely nothing red for ICMP Ping though in the Logs ...

EDIT 3: Issue seems to be when I setup CARP Outbound NAT for IPv4 -> IPv6 didn't have any. So I guess all traffic got silently dropped by OpenWRT ? Not sure (I tried to "Allow Invalid Traffic" etc, but that didn't help). As soon as I placed a "Outbound NAT" rule on OPNSense to match the CARP WAN IPv6 Address, then it started working.

But of course this means no IPv6 Public Address ...

EDIT 4: even with Outbound NAT for IPv6 it's EXCRUCIATING SLOW. Like 60 seconds before replying to Pings...

Right now everything is configured as Static IPv4/IPv6 for OPNSense (NOT the end-clients, i.e. GNU/Linux etc) in order to try to locate the Issue.

[Saarbremer]

even with Outbound NAT for IPv6 it's

<sound of me eating my desk> 

[luckylinux]

even with Outbound NAT for IPv6 it's

<sound of me eating my desk> 

Don't tell me ... I'm completely lost.

OPNSense logs don't show anything. OpenWRT logs don't show anything.

tcptraceroute6 2001:4860:4860::8888 (Google DNS Server IPv6 #1) works correctly although a bit slow
traceroute6 2001:4860:4860::8888 (Google DNS Server IPv6 #1) times out after Reaching OPNSense Router
ping -6 2001:4860:4860::8888 doesn't work at all.

It seems to be more of a "glitch" when it works, rather than the opposite.

Either NAT and/or Routing is completely broken.

EDIT 1: the Client is getting these Addresses via DHCP, but ONLY the /128 is being used (I can see it in OPNSense Logs).

Code: [Select]

eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet6 2aXX:XXX4:XXXX:1:XXXX:XXXX:a9f2:1858/128 scope global deprecated
       valid_lft forever preferred_lft 0sec

    inet6 2aXX:XXX4:XXXX:1:XXXX:XXXX:fe79:2d9a/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86388sec preferred_lft 14388sec

EDIT 2: the other thing I could imagine is that the Packet gets correctly "out" of OPNSense towards the OpenWRT Router, then either:
- Gets discarded/rejected (bad Source Address ?) - Nothing in the logs though
- OpenWRT doesn't know how to "give" the response back

Sticky Connections are enabled in OPNSense so that should take care of it (on OPNSense side). I also tried "Bind States to Interfaces" but it didn't make a difference.

Very different behavior between ICMP and TCPTRACEROUTE though ... TCP Seems to work correctly and now Google DNS Server is very quick to reply (using tcptraceroute6).

Which is weird considered all the Allow / Pass rules for ICMP ...

EDIT 3: Added a Static Route from OpenWRT -> OPNSense CARP IP as Gateway (I guess this is needed so that OpenWRT knows where to send the ICMP Packets back, I guess this is what Patrick tried to explain before).

So now Ping & Traceroute work. DNS Stopped working. I don't know why it's always 1 step forward 1 step backward all the time ...

EDIT 4: It seems to work now ... Both tcptraceroute6, traceroute6 -I (ICMP) / -S (TCP) / -U (UDP) work correctly, so does PING. The missing link was the Static Route from OpenWRT back to the LAN which is controlled by OPNSense.

I guess this is/was done automatically in the case of IPv4 with NAT+Sticky connections, so when OPNSense receives the packet back, it knows where to forward it to within the LAN.

Of course now that IPv6 Addresses are "direct" and the LAN is not managed by OpenWRT, there is no way for OpenWRT to know where to send the Packet to, since he doesn't know the Hosts. Thus adding the Static Route is probably what fixed it.

I still don't know however why OpenWRT Lost IPv6 all of a sudden and the only solution to get it back was to reboot the OpenWRT Device ...

EDIT 5: And Android also works with IPv6 now (although using SLAAC). Kinda annoying that you cannot see what IP Address those devices get ... Other than maybe running periodic ARP / Network Scans

[luckylinux]

OK ... I REALLY hope somebody could bring some clarity to the current status.

• OpenWRT Fiber Router is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.
• OPNSense Router1 is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.
• OPNSense Router2 is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.

On the other Hand, OPNsense Router DHCP Clients, lose IPv6 Ping "capability" after a while.

Nothing shows up in Firewall -> Log Files -> Live View (well, what shows is "Green"  ).

I'm suspecting an issue where the OPNsense CARP is misconfigured or misbehaving, like Packets are maybe going from LAN Client to OPNSense Router1, go to OpenWRT, then to the Internet, the reply comes back to OpenWRT, forwards to OPNSense CARP Adress and for some reason it goes to OPNSense Router2.

I'm not sure what to think otherwise .

No problem with IPv4, but IPv6 seems to last maximum a day, or as little as a few hours (didn't measure precisely ).

I wanted to use a Podman/Docker Container to monitor a Remote Host using IPv6 for this exact Reason, EXCEPT that Podman/Docker Rootless Containers don't really work well with IPv6 and external Routes to the Host.

So I cannot even monitor another HOST in the LAN with IPv6 at the moment from inside a Docker/Podman Container  .

[bimbar]

OK ... I REALLY hope somebody could bring some clarity to the current status.

• OpenWRT Fiber Router is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.
• OPNSense Router1 is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.
• OPNSense Router2 is able to ping IPv6 (testing with Google DNS Server 2001:4860:4860::8888) all the time.

On the other Hand, OPNsense Router DHCP Clients, lose IPv6 Ping "capability" after a while.

Nothing shows up in Firewall -> Log Files -> Live View (well, what shows is "Green"  ).

I'm suspecting an issue where the OPNsense CARP is misconfigured or misbehaving, like Packets are maybe going from LAN Client to OPNSense Router1, go to OpenWRT, then to the Internet, the reply comes back to OpenWRT, forwards to OPNSense CARP Adress and for some reason it goes to OPNSense Router2.

I'm not sure what to think otherwise .

No problem with IPv4, but IPv6 seems to last maximum a day, or as little as a few hours (didn't measure precisely ).

I wanted to use a Podman/Docker Container to monitor a Remote Host using IPv6 for this exact Reason, EXCEPT that Podman/Docker Rootless Containers don't really work well with IPv6 and external Routes to the Host.

So I cannot even monitor another HOST in the LAN with IPv6 at the moment from inside a Docker/Podman Container  .

You might want to debug packet flow using tcpdump on both opnsense firewalls.

Also, please post a complete network diagram with all interfaces and IPs, I can't really make sense of your setup.

[luckylinux]

Fair enough with the Network Diagram ... It's going to take a while -> DONE (see next post)

I also posted on OpenWRT Forum since I think it's more to do with some State Table / Firewall / Overload on the IPv6 Stack on their side:
https://forum.openwrt.org/t/ipv6-disappearing-after-a-while-for-lan-clients-troubleshooting-impossible-due-to-lack-of-openwrt-logs/198213

TLDR: once this Issue Occurs, restarting the WAN and WAN6 Interface on OpenWRT seems to solve the issue. Nothing else seemed to help (restarting LAN Client, restarting OPNSense Router1/Router2, restarting LAN Clients, etc).

Once that is done **on OpenWRT Router**, then the LAN Clients can once again Ping e.g. 2001:4860:4860::8888 (Google DNS Servers).

[luckylinux]

Attached Network Diagram.

For a bit added clarity:
- RED: segment of the IPv6 Delegation Prefix for WAN-interfaced Zones
- BLUE: segment of the IPv6 Delegation Prefix for LAN-interfaced Zones

[bimbar]

Attached Network Diagram.

For a bit added clarity:
- RED: segment of the IPv6 Delegation Prefix for WAN-interfaced Zones
- BLUE: segment of the IPv6 Delegation Prefix for LAN-interfaced Zones

This looks fine to me, provided the LAN clients get the CARP address as their gateway.

One thing perhaps, usually the gateway on LAN is a link-local address, so it should be preferred to use something like fe80::1 as CARP address on the LAN side and use that as gateway for the clients.
Clients like to talk to their routers using link-local addresses only.
The GUA CARP IP on LAN should then be unnecessary. It might well be that clients use the link-local addresses of the opnsense firewalls to route and then use the wrong one as their nexthop.

You should be able to find all of that out using tcpdump.