NAT64 (Tayga) + NAT-Outbound reply packets do not arrive (most of the time)

[cugerm]

Hello, to connect our IPv6-only local network (vlan2) to the IPv4 internet, we configured NAT64 as described in the Tayga tutorial. But at first the connection did not work. Tcpdump shows an almost functioning routing path, but from the last "NAT64 hop" the packet is not forwarded back to vlan2. After hours, however, it suddenly worked, even though we had not made any configuration changes (we only changed logging settings during that time, but that does not seem to be related, we could not reproduce the behavior)... Here the excerpt from the tcpdump traces:

Code: [Select]

# THE ENVIRONMENT
foo.example.org = 1.2.3.4 = A public server with only an IPv4 address
14141 = TCP Test Port that is listening on 1.2.3.4
64:ff9b::aaaa:bbbb = the (anonymised) IPv6 translation of 1.2.3.4 address
2001:db8:0000:...  = the (anonymised) IPv6 prefix (assigned by ISP)
2001:db8:0000:2::1 = OPNsense incl. Tayga and Unbound DNS
100.100.100.136/29 = the (anonymised) public IPv4 network (assigned by ISP)


# CONNECTING FROM VLAN2 HOST
# server1 (2001:db8:0000:2::10, vlan2)
$ nc foo.example.org 14141
<hangs until timeout>


# TCPDUMPS AT OPNSENSE
opnsense$ tcpdump -n -i bge0_vlan2
IP6 2001:db8:0000:2::10.52218 > 2001:db8:0000:2::1.53: 45473+ A? foo.example.org. (35)
IP6 2001:db8:0000:2::10.52218 > 2001:db8:0000:2::1.53: 44454+ AAAA? foo.example.org. (35)
IP6 2001:db8:0000:2::1.53 > 2001:db8:0000:2::10.52218: 45473 1/0/0 A 1.2.3.4 (51)
IP6 2001:db8:0000:2::1.53 > 2001:db8:0000:2::10.52218: 44454 1/0/0 AAAA 64:ff9b::aaaa:bbbb (63)

opnsense$ tcpdump -n -i bge0_vlan2
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S],...

opnsense$ tcpdump -n -i nat64
IP6 2001:db8:0000:2::10.51314 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
IP 10.64.120.219.51314 > 1.2.3.4.14141: Flags [S], ...

opnsense$ tcpdump -n -i bnxt1 port 14141   # WAN interface
IP 100.100.100.140.38343 > 1.2.3.4.14141: Flags [S], ...
IP 1.2.3.4.14141 > 100.100.100.140.38343: Flags [S.], ...

opnsense$ tcpdump -n -i nat64
IP 1.2.3.4.14141 > 10.64.120.219.51314: Flags [S.], ...
IP6 64:ff9b::aaaa:bbbb.14141 > 2001:db8:0000:2::10.51314: ...

opnsense$ tcpdump -n -i bge0_vlan2
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
As you can see: NAT64 sends "64:ff9b::aaaa:bbbb.14141 > 2001:db8:0000:2:10" but it does not reach the bge0_vlan2 interface (or any other, we checked all). Firewall rules do not block the connection, despite activating the entire logging (incl. default rules), the log does not show any related entries. And as I wrote: After hours it suddenly worked and the final vlan2 tcpdump showed the packages and the nc command got the response from 1.2.3.4's listening nc. But since we had little confidence in the situation, we restarted the opnsense server and unfortunately the old state was immediately restored: The connection failed again. So, we have the following questions and would appreciate your help:

• Are there obvious configuration errors? We have attached our settings below.
• Since there seems to be a time dependency: Are there any caches/buffers that we should flush?
• Is it normal that the nat64 interface has only an ipv4 but not an ipv6 address?
• Are there (FreeBSD) command/tools we can use for deeper troubleshooting?

Thank you!!!


Our setup:

Code: [Select]

##### ISP Gateway:
Static IP 2001:db8:0000::1
Upstream Gateway = Yes
Disable Gateway Monitoring = Yes
Disable reply-to on WAN rules  = No   # also tried Yes with same result


##### Tayga and Unbound DNS:
IPv4 Address                            10.64.0.1
IPv4 NAT64 Interface Address    10.65.64.1
IPv6 Address                            2001:db8:0000:5001:64::1
IPv6 NAT64 Interface Address    2001:db8:0000::4
IPv6 Prefix                                64:ff9b::/96
IPv4 Pool                                  10.64.0.0/16
Enable DNS64 Support =           Yes
DNS64 Prefix = Not set             # to use default 64:ff9b::/96


##### NAT 64 interface and Routing Table:
opnsense$ ifconfig nat64
nat64: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet 10.65.64.1 --> 10.64.0.1 netmask 0xffffffff
groups: tun
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 85229

opnsense$ netstat -rn
Destination        Gateway            Flags     Netif Expire
default            100.100.100.137     UGS       bnxt1
10.64.0.0/16       link#16            US        nat64
10.64.0.1          link#16            UH        nat64
10.65.64.1         link#16            UHS         lo0
100.100.100.136/29 link#4             U         bnxt1
100.100.100.140    link#4             UHS         lo0
127.0.0.1          link#5             UH          lo0

Internet6:
Destination               Gateway            Flags     Netif Expire
default                   2001:db8:0000::1   UGS       bnxt1
::1                       link#5             UHS         lo0
64:ff9b::/96              link#16            US        nat64
2001:db8:0000::/48        link#4             U         bnxt1
2001:db8:0000::4          link#4             UHS         lo0
2001:db8:0000:1::/64      link#10            U      bge0_vlan1
2001:db8:0000:1::1        link#10            UHS         lo0
2001:db8:0000:2::/64      link#9             U      bge0_vlan2
2001:db8:0000:2::1        link#9             UHS         lo0
2001:db8:0000:3::/64      link#11            U      bnxt0_vlan3
2001:db8:0000:3::1        link#11            UHS         lo0


##### pf/NAT rules:
Tayga Interface:
  pass IPv4 from 10.64.0.0/16 to any  # also tried any to any with same result
Vlan 2 Interface:
  pass IPv6 from any to any
NAT Outbound:
  IPv4, Source = 10.64.0.0/16, Destination = any, Translation = WAN Address (100.100.100.140)

[cugerm]

Anyone? Perhaps tips for further debugging? Is there any (network interface, memory, buffer, cache, ... related) command that shows what happens to the packet after leaving the nat64 interface? Or kind of command/filter that allows to observe the packages while traveling through all interfaces? Thanks again + Best

[cugerm]

After a more detailed analysis in the last few days we realized that this setence from the Tayga plugin documentation (https://docs.opnsense.org/manual/how-tos/tayga.html) is our problem:

IPv6 NAT64 Interface Address: ... For simplicity, you may reuse an address of another OPNsense interface.

Looking at the start script /usr/local/etc/rc.d/opnsense-tayga you can see, that it tries to configure the IPv6 NAT64 Interface Address on the nat64 interface. If the IP is already used by another interface, this configuration will fail and ifconfig nat64 shows only an IPv4 but not an IPv6 address. The result ist, that tayga will not work (completely) as reported. Changing the IPv6 NAT64 Interface Address leads to a stable working in our environment.

[Maurice]

Thanks for your detailed analysis. I was able to reproduce this behavior in OPNsense 22.1. Something must have changed in the interface code since I wrote the how-to almost two years ago. I'm 99% sure I tested that reusing an existing interface address actually worked back then (in OPNsense 20.1).

I'll create a PR to remove this sentence from the documentation.

Cheers

Maurice

https://github.com/opnsense/docs/pull/391

[cugerm]

Great, thank you for the documentation and the PR. Perhaps it is also worth mentioning that an ULA IPv6 address can be used. This is a working configuration:

Code: [Select]

IPv4 Address: 10.14.0.1                         # from IPv4 Pool
IPv4 NAT64 Interface Address: 10.15.0.1         # not from IPv4 Pool
IPv6 Address: <GUA IPv6 address>                # one of your "public" IPv6 adresses
IPv6 NAT64 Interface Address: fd00:14::1        # not used by other interfaces
IPv6 Prefix: 64:ff9b::/96                       # well-known prefix
IPv4 Pool: 10.14.0.0/16
Don't forget DNS64 configuration and firewall/nat rules as documented by Maurice.

[Maurice]

Correct, Tayga has no limitations regarding ULAs. I also use ULAs for the NAT64 interface in my setups.