Native NAT64 support

[bestboy]

Hi,

I'm currently experimenting with IPv6-only networks. I might soon get a DS-Lite connection and consider finally getting rid of IPv4 (except for some niches) and run an IPv6-preferred network.
For testing I have NAT64 set up with the Tayga plugin and DNS64 deployed via Unbound. This setup works ok. The fact that many clients on Android, iOS and recent Windows support 464XLAT by now helps to mitigate issues with IPv4 literals that can not be addressed by DNS64. DHCP option 108 is already available and PREF64 support is on the way (I think radvd 2.20 supports it already, only UI support is missing). All things considered, I believe NAT64 could be a viable IPv4 transitioning solution. There is really only one single issue with my tests: IPSec

Why is IPSec important? Well, other than the fact that most corporate VPN solutions use IPSec, the main reason IMHO is VOWIFI aka Wifi Calling. VOWIFI allows handheld devices to extend the mobile network via a WIFI network. This is achieved by establishing an IPSec connection to an Evolved Packet Data Gateway (ePDG) of the mobile provider via the internet. This essentially brings mobile services to where no mobile reception is. I'm sure most of you know Wifi Calling. IMHO a killer feature not to be missed.
The IPSec mode used by the VPN connection to the ePDG is tunnel mode with the ESP protocol and UDP encapsulation (NAT-T). Unfortunately, the ePDGs seem to be using IPv4 exclusively (see https://www.netify.ai/resources/mobile-gateways). I do not know why there is still no support for IPv6, even though mobile providers feel the pain of IPv4 address space running out. I also do not know of any plans to change that in the near future. So one can assume Wifi Calling with IPv6 will require NAT64 for the time being.

Now, the good thing is, NAT64 in general should be able to support IPSec with the ESP protocol (unlike IPSec with the AH protocol). However, NAT64 via Tayga does not. In my tests I was not able to use IPSec and in extension Wifi Calling. As far as I can tell Tayga simply does not have support for protocol 50 (ESP) - only 1 (ICMP), 6 (TCP) and 17 (UDP). I believe support for rewriting ESP headers could just be added, but it's not there for now.

TL;DR: NAT64 via Tayga is incomplete.

Which brings me to the actual question: What is the current state of "native" NAT64 support on OpnSense? Are there any plans to support NAT64 besides the Tayga plugin? I could not find anything in the roadmap about it.
I'd assume using Jool is out of the question being a Linux kernel module.
FreeBSDs own IPFW has support for NAT64. The original PF on OpenBSD has support for NAT64, too. The FreeBSD port of PF however does not have NAT64 support AFAIK, because it was forked before NAT64 support was added.
PFSense seems to have gotten native support for NAT64 recently, but I do not know how it is implemented there.

[Monviech (Cedrik)]

Wouldn't it still be better to run dual stack for a few more years until IPv6 penetration reached the last isolated IPv4-only services? Dual stack is well documented and just works for most setups without much oddities.

Im just curious why running IPv6 only with a translation to IPv4 is even worth it. I don't negate the hobbyist aspect of it though.

Though if the tools do not offer what you need, why not wait for a while longer?

[Patrick M. Hausen]

Im just curious why running IPv6 only with a translation to IPv4 is even worth it.

We do it in our hosting environment to preserve IPv4 addresses. Standard product is a FreeBSD jail with IPv6 only, NAT64 and DNS64 so the customer can access e.g. Github. Inbound SNI proxy for HTTP and HTTPS, one proxy per physical machine. We also provide a dual stack SSH jumphost.

NAT64 is done with IPFW.

IPv4 addresses cost real money, nowadays. If you want dual stack for your jail, we charge 5€ per month.

[patient0]

The FreeBSD port of PF however does not have NAT64 support AFAIK, because it was forked before NAT64 support was added.
PFSense seems to have gotten native support for NAT64 recently, but I do not know how it is implemented there.

Native NAT64 - by the keyword 'af-to' - was introduced with FreeBSD 15[1].

The question came up before on the forum but I can't find it right now. The gist is: OPNsense will adopt FreeBSD 15.1, timeline Mid to End 2026, if I remember correctly (or Dec 2026 - Mid 2027?).

Btw, apalrd has taken over Tayga[2] and is pushing it further but I'm not sure if his/their effort is Linux only. In the video I think is only Linux mentioned.

[1]https://man.freebsd.org/cgi/man.cgi?query=pf.conf&manpath=FreeBSD+15.0-RELEASE
[2]https://github.com/apalrd/tayga
[3]https://www.youtube.com/watch?v=WlQH8KubgiA

[Maurice]

I've experimented with WiFi Calling over NAT64 in the past and never got it to work. If anyone had success with this (with any NAT64), I'd be interested. But I'd say it's up to the MNOs to add IPv6 support to their ePDGs.

The new pf 'af-to' feature in FreeBSD 15 indeed makes native NAT64 in OPNsense realistic in the near future. From a quick look, it seems that's what pfSense uses, too (by using FreeBSD prerelease code?). But I'm not very confident this solves the WiFi Calling use case (although I'd be happy to be proven wrong).

@Monviech For general purpose networks, directly switching from Dual Stack to IPv6-only without NAT64 seems unrealistic anytime soon. Some IPv4-only services will be around for many, many years. So IPv6-only with NAT64 is a sensible intermediate step. Among other reasons, it eliminates the management and troubleshooting overhead of Dual Stack while maintaining compatibility with legacy IP.
If you're more conservative, you can go Dual Stack with IPv6-only preferred first, but this requires NAT64, too.

@patient0 apalrd has mentioned being an OPNsense user, but I'm not sure whether he uses the OPNsense Tayga plugin or Tayga on a dedicated Linux system.

Cheers
Maurice

[bestboy]

Thanks everybody for the useful input.

I want to address a few questions and comments.

Wouldn't it still be better to run dual stack for a few more years until IPv6 penetration reached the last isolated IPv4-only services? Dual stack is well documented and just works for most setups without much oddities.

Im just curious why running IPv6 only with a translation to IPv4 is even worth it. I don't negate the hobbyist aspect of it though.

Though if the tools do not offer what you need, why not wait for a while longer?

Well, yes and no.
I'm currently running Dual Stack and yes it works. I have 2 issues with DS:

• Running DS is much more complicated than just running IPv6 only. IPv6-only is kinda simple really. Running DS is also much more complicated than just running IPv4 only. IPv4 is what most (hobby) admins are grown up with and are using since day 1.
  But running both IPv4 and IPv6 in DS requires extra knowledge and efforts. And I feel that the DS overhead has become a boon of IPv6 adoption by now. Most hobby admins probably stick with IPv4 coz that's what they know - unless they are enthusiasts.
• Running DS is no longer that easy. Many ISPs do no longer offer DS, because they lack the IPv4 addresses. Handing out precious IPv4 to retail customers is getting rare and is only an option to old "ISP nobility" which has been around since the early days of the internet and thus owning enough IPv4 address space. Most newer ISPs just cannot provide IPv4 to retail customers. It's not even an option, if you are willing to pay extra. What you get instead is DS-Lite. DS-Lite gives you a proper global IPv6 prefix and a CGNAT IPv4 address in 100.64.0.0/16 space. This private IPv4 is then NATed by the ISP with NAT44 to a few public IPv4 addresses it owns. CGNAT is the worst IMHO. Coming from DS, switching to DS-Lite is a big step backwards. Personally I think it might be more beneficial to "flight forward" in this situation, forget about the crappy CGNAT IPv4 and just use IPv6 only. I'd happily trade CGNAT for NAT64

But I'd say it's up to the MNOs to add IPv6 support to their ePDGs.

Yes, I agree that would be the best solution. But I believe there must be some obstacle preventing it even though they try to adopt IPv6 fast. I do not know much about IPSec, but I heard them say "Keep your IPv4 and your IPv6 IPSec separate". If this is true, then they need to run the entire ePDG tunneling twice over. Yet another example of why DS is cumbersome and avoided.

I've experimented with WiFi Calling over NAT64 in the past and never got it to work. If anyone had success with this (with any NAT64), I'd be interested.

Ok, that's a bummer. I was assuming the native NAT64 solutions may support ESP. Which NAT64 implementation were you using?

Again, I do not know much about IPSec, but in ESP tunnel mode I believe there should be 2 IP headers with addresses to be exchanged. An inner one of the original packet that is to be tunneled, and an outer one that is added by IPSec for the sake of encapsulation and transport. I'd assume that the outer header is typically NATed ok, but the inner one is probably omitted, if there is no explicit support for ESP.

[Monviech (Cedrik)]

What about UDP encapsulated ESP, is that supported? The OPNsense is able to do that for IPv6 since a kernel patch last year (disclaimer, if the opnsense itself is the ipsec peer) and its a standard for IKEv2 and required on most mobile devices like Android (so why is pure esp used for wifi calling)

UDP packets should match with what Tayga expects I suppose?

[Maurice]

Which NAT64 implementation were you using?

As far as I remember, Tayga and Jool. I most certainly performed some packet captures and seem to recall that the phone didn't even try to use IPv6 for VoWiFi. There wasn't anything after the DNS lookup for the ePDG. I suspected that the VoWiFi stack of my phone didn't even support IPv6.

But it's been a while and the details are a bit foggy, so you might want to repeat these tests.

@Monviech There are zero user configurable options for VoWiFi, other than on and off. That's pretty much a black box deep in the phone's software stack. I seem to remember that the core functionality is even provided by the SoC, not Android itself.

[Patrick M. Hausen]

I wonder what Tayga might do that three trivial IPFW rules [1] don't. The latter is all we use to implement NAT64 in our DC. Combine with Unbound DNS64, done.

[1]

Code Select Expand

nat64lsn NAT64 create prefix4 <my external IPv4>/32
add 5000 nat64lsn NAT64 ip from <my IPv6 prefix> to 64:ff9b::/96 in
add 5100 nat64lsn NAT64 ip from any to <my external IPv4>/32 in


[Maurice]

There are many nuances in NAT64, e. g. how exactly ICMP messages are translated between versions (which is quite complex because the payload has to be translated, too).

apalrd (new maintainer of Tayga) did a comparison of different NAT64 implementations:

https://youtu.be/WlQH8KubgiA

[overbored]

I was looking into the same thing regarding VoWifi, as that was my only issue when trying to go full "IPv6 preferred". Turns out Tayga does support the RFC (7915) that handles these 0 length checksum fields.

https://github.com/apalrd/tayga/pull/76

A value of `udp-cksum-mode fwd` or `udp-cksum-mode calc` in your `tayga.conf` should fix the issue, assuming all else is in place. As soon as I did this, it started working locally (on my iPhone at least).

[Maurice]

Thanks for making this connection, @overbored!

I performed a packet capture again. My phone makes no attempt to establish the VoWiFi tunnel over IPv6.
It only performs a DNS query for the ePDG's A record, not AAAA. Then there's absolutely nothing after the DNS reply.
It doesn't matter whether I filter A records in Unbound or not (AAAA-only mode).
It really seems that my phone's VoWiFi stack is strictly IPv4-only and doesn't attempt to use Android's CLAT either.

This might be different with other / newer phones like your iPhone.

@bestboy, would you mind testing this Tayga setting with your phone?

Cheers
Maurice

[apalrd]

A few notes (as the current dev of Tayga):

- Tayga does pass all IP protocols which are not TCP/UDP/ICMP*, but it only does checksum adjustments to those two. So it will still pass ESP (proto 50) as-is.
* except protocols which are specific to an IP version - receiving proto 58 (ICMPv6) over IPV4 will be dropped, for example

- ESP alone cannot be NATTed. The entire ESP header is authenticated, so NAT on-path cannot modify it. No NAT translation RFC for ESP exists. This is why Tayga does not implement anything special for ESP.

- ESP should be encapsulated in UDP to traverse NAT. I assume then that your carrier is doing this, so the lack of special ESP handling in Tayga is a red herring anyway.

- If it's a case of UDP packets with zero checksums, Tayga originally dropped all UDP packets with a zero checksum, which was a compliant behavior per RFC 6145. Tayga now has a configuration option to select drop, calculate, or forward. The version (0.9.5) in OPNsense supports this, but it would need `udp-cksum-mode fwd` in the tayga.conf to enable. 'fwd' is not RFC compliant, btw, but it basically punts the handling of udp fragments to whoever does NAT44 (pf, in our case), which is probably the best option. 

- `af-to` was added to pf in FreeBSD by Netgate, and is used in pfSense. FreeBSD ipfw has had nat64 support for a few versions now, but OPNsense (and pfSense) have not used it afaik due to both of them using pf instead of ipfw.

- Tayga fully supports both Linux and FreeBSD

- I personally do use Tayga on OPNsense for NAT64 at home and have for a few years, although I also run Tayga on Linux for development.

- There's a new release of Tayga from just a few days ago, but it does not change anything relevant here

[Maurice]

Welcome to the forum, apalrd! Nice to see you here. And thanks for the clarifications.

Would it make sense to change the default udp-cksum-mode? Either in Tayga itself or in the OPNsense plugin.
While this won't magically solve all IPsec related issues, it seems to unbreak some real-world use cases (as reported by @overbored).

Cheers
Maurice

(author of the OPNsense Tayga NAT64 how-to)

[apalrd]

Hello! Nice to see you too!

Tayga's default is the same behavior that Tayga used to do before the configuration was added, but it's also the only behavior which is not optional to implement per RFCs. For this specific case, it would seem that 'fwd' is the best approach for OPNsense, but that is also not RFC compliant, although it's probably the most correct behavior.

Per RFC2460*, IPv6 requires UDP packets to have checksums, although they were optional in IPv4. IPv6 has no checksum, so it relies on the transport layer checksum which includes an IP psuedo-header (for v4 and v6). Arguably, none of these checksums are very good anyway, since IPv4, UDP, and TCP all use the 'Internet Checksum', which is just a one's complement 16-bit sum of words, requires end-around carries on modern two's complement hardware, and is laughably bad at detecting more than the simplest errors and cannot detect mis-ordered words (since they add up to the same number). But it is what has been used for decades.

Option 1 - drop - this drops packets 4->6 which have no UDP checksum, since they would not be valid in IPv6. This impacts protocols which are cheating in IPv4 by not generating checksums, which includes RFC3948 (UDP encapsulation of ESP), which is why we are in this scenario [1].

Option 2 - calc - this calculate a UDP checksum for 4->6 packets, which seems great, except for fragments. Since Tayga does not reassemble fragments (IPv4 fragments get translated into IPv6 fragments, which do exist and are legal but not commonly used), we cannot correctly calculate the checksum of a packet which comes in fragmented. ESP-in-UDP may not hit this case, but I'm guessing it's more likely to suffer from misconfigured MTU within the tunnel which would cause the tunnel overhead to require fragmentation. Tayga can also fragment a packet if it's too large on the IPV6 side but not IPv4 side, but that is done after the checksum would be calculated and is fine. If the packets are fragmented, calc turns into drop. Calc also has a performance impact, 16-bit one's complement addition is not very well optimized, and Tayga is already single-threaded and not very performant.

Option 3 - forward - this forwards the zero-checksum packets as-is, assuming the receiver is compliant enough to not care. If the receiver is doing 6->4 translation anyway, then the receiver must accept it, but if it's doing native IPV6 it may not. This is not an option that RFC7915 [2] allows, however, RFC8200 (IPv6) [3] updated RFC2460 (IPv6) to allow zero checksums for tunneling protocols based on the work done by RFC6936 [4].

Given all of this, I suggest OPNsense set tayga to use 'fwd'

[1] - https://datatracker.ietf.org/doc/html/rfc3948#section-2.1
[2] - https://datatracker.ietf.org/doc/html/rfc7915#section-4.5
[3] - https://datatracker.ietf.org/doc/html/rfc8200#section-8.1
[4] - https://datatracker.ietf.org/doc/html/rfc6936
*RFC8200 changed this guidance to carve-out for ESP-in-UDP, basically