1
22.1 Legacy Series / NAT64 (Tayga) + NAT-Outbound reply packets do not arrive (most of the time)
« on: March 16, 2022, 10:43:14 pm »
Hello, to connect our IPv6-only local network (vlan2) to the IPv4 internet, we configured NAT64 as described in the Tayga tutorial. But at first the connection did not work. Tcpdump shows an almost functioning routing path, but from the last "NAT64 hop" the packet is not forwarded back to vlan2. After hours, however, it suddenly worked, even though we had not made any configuration changes (we only changed logging settings during that time, but that does not seem to be related, we could not reproduce the behavior)... Here the excerpt from the tcpdump traces:
As you can see: NAT64 sends "64:ff9b::aaaa:bbbb.14141 > 2001:db8:0000:2:10" but it does not reach the bge0_vlan2 interface (or any other, we checked all). Firewall rules do not block the connection, despite activating the entire logging (incl. default rules), the log does not show any related entries. And as I wrote: After hours it suddenly worked and the final vlan2 tcpdump showed the packages and the nc command got the response from 1.2.3.4's listening nc. But since we had little confidence in the situation, we restarted the opnsense server and unfortunately the old state was immediately restored: The connection failed again. So, we have the following questions and would appreciate your help:
Thank you!!!
Our setup:
Code: [Select]
# THE ENVIRONMENT
foo.example.org = 1.2.3.4 = A public server with only an IPv4 address
14141 = TCP Test Port that is listening on 1.2.3.4
64:ff9b::aaaa:bbbb = the (anonymised) IPv6 translation of 1.2.3.4 address
2001:db8:0000:... = the (anonymised) IPv6 prefix (assigned by ISP)
2001:db8:0000:2::1 = OPNsense incl. Tayga and Unbound DNS
100.100.100.136/29 = the (anonymised) public IPv4 network (assigned by ISP)
# CONNECTING FROM VLAN2 HOST
# server1 (2001:db8:0000:2::10, vlan2)
$ nc foo.example.org 14141
<hangs until timeout>
# TCPDUMPS AT OPNSENSE
opnsense$ tcpdump -n -i bge0_vlan2
IP6 2001:db8:0000:2::10.52218 > 2001:db8:0000:2::1.53: 45473+ A? foo.example.org. (35)
IP6 2001:db8:0000:2::10.52218 > 2001:db8:0000:2::1.53: 44454+ AAAA? foo.example.org. (35)
IP6 2001:db8:0000:2::1.53 > 2001:db8:0000:2::10.52218: 45473 1/0/0 A 1.2.3.4 (51)
IP6 2001:db8:0000:2::1.53 > 2001:db8:0000:2::10.52218: 44454 1/0/0 AAAA 64:ff9b::aaaa:bbbb (63)
opnsense$ tcpdump -n -i bge0_vlan2
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S],...
opnsense$ tcpdump -n -i nat64
IP6 2001:db8:0000:2::10.51314 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
IP 10.64.120.219.51314 > 1.2.3.4.14141: Flags [S], ...
opnsense$ tcpdump -n -i bnxt1 port 14141 # WAN interface
IP 100.100.100.140.38343 > 1.2.3.4.14141: Flags [S], ...
IP 1.2.3.4.14141 > 100.100.100.140.38343: Flags [S.], ...
opnsense$ tcpdump -n -i nat64
IP 1.2.3.4.14141 > 10.64.120.219.51314: Flags [S.], ...
IP6 64:ff9b::aaaa:bbbb.14141 > 2001:db8:0000:2::10.51314: ...
opnsense$ tcpdump -n -i bge0_vlan2
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
IP6 2001:db8:0000:2::10.51300 > 64:ff9b::aaaa:bbbb.14141: Flags [S], ...
As you can see: NAT64 sends "64:ff9b::aaaa:bbbb.14141 > 2001:db8:0000:2:10" but it does not reach the bge0_vlan2 interface (or any other, we checked all). Firewall rules do not block the connection, despite activating the entire logging (incl. default rules), the log does not show any related entries. And as I wrote: After hours it suddenly worked and the final vlan2 tcpdump showed the packages and the nc command got the response from 1.2.3.4's listening nc. But since we had little confidence in the situation, we restarted the opnsense server and unfortunately the old state was immediately restored: The connection failed again. So, we have the following questions and would appreciate your help:
- Are there obvious configuration errors? We have attached our settings below.
- Since there seems to be a time dependency: Are there any caches/buffers that we should flush?
- Is it normal that the nat64 interface has only an ipv4 but not an ipv6 address?
- Are there (FreeBSD) command/tools we can use for deeper troubleshooting?
Thank you!!!
Our setup:
Code: [Select]
##### ISP Gateway:
Static IP 2001:db8:0000::1
Upstream Gateway = Yes
Disable Gateway Monitoring = Yes
Disable reply-to on WAN rules = No # also tried Yes with same result
##### Tayga and Unbound DNS:
IPv4 Address 10.64.0.1
IPv4 NAT64 Interface Address 10.65.64.1
IPv6 Address 2001:db8:0000:5001:64::1
IPv6 NAT64 Interface Address 2001:db8:0000::4
IPv6 Prefix 64:ff9b::/96
IPv4 Pool 10.64.0.0/16
Enable DNS64 Support = Yes
DNS64 Prefix = Not set # to use default 64:ff9b::/96
##### NAT 64 interface and Routing Table:
opnsense$ ifconfig nat64
nat64: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet 10.65.64.1 --> 10.64.0.1 netmask 0xffffffff
groups: tun
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 85229
opnsense$ netstat -rn
Destination Gateway Flags Netif Expire
default 100.100.100.137 UGS bnxt1
10.64.0.0/16 link#16 US nat64
10.64.0.1 link#16 UH nat64
10.65.64.1 link#16 UHS lo0
100.100.100.136/29 link#4 U bnxt1
100.100.100.140 link#4 UHS lo0
127.0.0.1 link#5 UH lo0
Internet6:
Destination Gateway Flags Netif Expire
default 2001:db8:0000::1 UGS bnxt1
::1 link#5 UHS lo0
64:ff9b::/96 link#16 US nat64
2001:db8:0000::/48 link#4 U bnxt1
2001:db8:0000::4 link#4 UHS lo0
2001:db8:0000:1::/64 link#10 U bge0_vlan1
2001:db8:0000:1::1 link#10 UHS lo0
2001:db8:0000:2::/64 link#9 U bge0_vlan2
2001:db8:0000:2::1 link#9 UHS lo0
2001:db8:0000:3::/64 link#11 U bnxt0_vlan3
2001:db8:0000:3::1 link#11 UHS lo0
##### pf/NAT rules:
Tayga Interface:
pass IPv4 from 10.64.0.0/16 to any # also tried any to any with same result
Vlan 2 Interface:
pass IPv6 from any to any
NAT Outbound:
IPv4, Source = 10.64.0.0/16, Destination = any, Translation = WAN Address (100.100.100.140)