"
I have exhausted all my troubleshooting, someone please tell me why I'm stupid.
I have DNS64 setup correctly, clients are resolving the virtual IPv6 address, however they have been timing out and falling back to ipv4 (when they have an IPv4 address).
Tayga config:
IPv4 Address 192.168.255.1
IPv4 NAT64 Interface Address 172.20.0.1
IPv6 Address _blank_
IPv6 NAT64 Interface Address fd01::a:172:20:0:1
IPv6 Prefix 64:ff9b::/96
IPv4 Pool 192.168.255.0/24
"Only used for ICMP."
This phrase in the tayga config is doing *a lot* of heavy lifting. (It makes it seem like its essentially useless if you dont care about icmp, which for a pure netcat TCP test, I dont)
NAT Oubound rule:
Interface Source Source Port Destination Destination Port NAT Address NAT Port Static Port Description
WAN 192.168.255.0/24 * * * Interface address * NO NAT64 Tayga Outbound NAT
Tayga Interface rule (allow all):
Pass IN Tayga IPv4+IPv6 * * * * * *
Looks like I've hit all the points in the setup wiki https://docs.opnsense.org/manual/how-tos/tayga.html
And from my troubleshooting below, it seems like the outbound nat, firewall rule, and tayga itself are all operating properly.
I think I have it narrowed down to the internal IPv6 return traffic being dropped by the kernel.
My tcpdumps are showing:
WAN Interface
[root@EFW ~]# tcpdump -vvvniigb0 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:01:18.926683 IP (tos 0x0, ttl 88, id 25707, offset 0, flags [none], proto TCP (6), length 52)
publicIPv4.42260 > 4.79.142.200.443: Flags [ S ], cksum 0x35d7 (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.960649 IP (tos 0x0, ttl 121, id 16200, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0xf1e9 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965648 IP (tos 0x0, ttl 121, id 17602, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0xf1e9 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:27.971536 IP (tos 0x0, ttl 121, id 18655, offset 0, flags [DF], proto TCP (6), length 48)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0x25f9 (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:39.967678 IP (tos 0x0, ttl 121, id 20842, offset 0, flags [DF], proto TCP (6), length 40)
4.79.142.200.443 > publicIPv4.42260: Flags [R], cksum 0x52c9 (correct), seq 1881125065, win 0, length 0
5 packets captured
4058 packets received by filter
0 packets dropped by kernel
Tayga Interface
[root@EFW ~]# tcpdump -vvvninat64 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on nat64, link-type NULL (BSD loopback), snapshot length 262144 bytes
18:01:18.926619 IP6 (flowlabel 0x6a58e, hlim 90, next-header TCP (6) payload length: 32) fd01::1:172:20:20:10.39346 > 64:ff9b::44f:8ec8.443: Flags [ S ], cksum 0xff4d (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.926638 IP (tos 0x0, ttl 89, id 25707, offset 0, flags [none], proto TCP (6), length 52)
192.168.255.195.39346 > 4.79.142.200.443: Flags [ S ], cksum 0x3da6 (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.960684 IP (tos 0x0, ttl 120, id 16200, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0xf9b8 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:18.960718 IP6 (hlim 119, next-header TCP (6) payload length: 32) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xbb60 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965711 IP (tos 0x0, ttl 120, id 17602, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0xf9b8 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965742 IP6 (hlim 119, next-header TCP (6) payload length: 32) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xbb60 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:27.971559 IP (tos 0x0, ttl 120, id 18655, offset 0, flags [DF], proto TCP (6), length 48)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0x2dc8 (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:27.971590 IP6 (hlim 119, next-header TCP (6) payload length: 28) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xef6f (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:39.967745 IP (tos 0x0, ttl 120, id 20842, offset 0, flags [DF], proto TCP (6), length 40)
4.79.142.200.443 > 192.168.255.195.39346: Flags [R], cksum 0x5a98 (correct), seq 1881125065, win 0, length 0
18:01:39.967789 IP6 (hlim 119, next-header TCP (6) payload length: 20) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [R], cksum 0x1c40 (correct), seq 1881125065, win 0, length 0
10 packets captured
10 packets received by filter
0 packets dropped by kernel
Internal Interface
[root@EFW ~]# tcpdump -vvvnivlan01 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on vlan01, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:01:18.926584 IP6 (flowlabel 0x6a58e, hlim 91, next-header TCP (6) payload length: 32) fd01::1:172:20:20:10.39346 > 64:ff9b::44f:8ec8.443: Flags [ S ], cksum 0xff4d (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
1 packet captured
2694 packets received by filter
0 packets dropped by kernel
I think this one counter `failures of source address selection` is the symptom, as it tends to increase ~60seconds after each test
[root@EFW ~]# netstat -s -p ip6
ip6:
34443359 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
276 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 atomic fragments
0 packets reassembled ok
1864797 packets for this host
31895889 packets forwarded
0 packets not forwardable
0 redirects sent
3043173 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
2 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
4 packets that violated scope rules
62 multicast packets which we don't join
Input histogram:
hop by hop: 699
TCP: 32210783
UDP: 1981356
ICMP6: 250233
PIM: 12
Mbuf statistics:
16835632 one mbuf
two or more mbuf:
lo0= 2124
wg1= 377865
17227738 one ext mbuf
0 two or more ext mbuf
0 packets whose headers are not contiguous
0 tunneling packets that can't find gif
0 packets discarded because of too many headers
2648 failures of source address selection
source addresses on an outgoing I/F
53783 link-locals
77070 globals
source addresses on a non-outgoing I/F
82 globals
2648 addresses scope=0xf
source addresses of same scope
53780 link-locals
77152 globals
source addresses of a different scope
3 link-locals
Source addresses selection rule applied:
130935 first candidate
15095 same address
53708 appropriate scope
48685 outgoing interface
82 matching label
42921 longest match
Fw info: (OPNsense 26.1.8_5-amd64)
routes:
[root@EFW ~]# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default publicipv4gw UGS igb0
1.1.1.1 publicipv4gw UGHS igb0
10.10.0.0/16 172.20.0.2 UGS ixl2
publicipv4block/24 link#1 U igb0
publicipv4 link#14 UHS lo0
127.0.0.1 link#14 UH lo0
172.20.0.0/30 link#12 U ixl2
172.20.0.1 link#14 UHS lo0
172.20.19.0/29 link#5 U igb4
172.20.19.1 link#14 UHS lo0
172.20.20.0/24 link#18 U vlan01
172.20.20.1 link#14 UHS lo0
172.20.21.4/30 link#2 U igb1
172.20.21.5 link#14 UHS lo0
172.20.22.0/24 link#19 U vlan02
172.20.22.1 link#14 UHS lo0
172.20.24.0/24 link#20 U vlan03
172.20.24.1 link#14 UHS lo0
172.20.253.0/30 link#22 U wg1
172.20.253.1 link#14 UHS lo0
192.168.12.0/24 link#4 U igb3
192.168.12.1 link#14 UHS lo0
192.168.255.0/24 link#24 US nat64
192.168.255.1 link#24 UH nat64
Internet6:
Destination Gateway Flags Netif Expire
default fe80::256:2bff:fe76:b022%igb0 UG igb0
::1 link#14 UHS lo0
64:ff9b::/96 link#24 US nat64
publicipv6 link#14 UHS lo0
publicipv6prefix::/60 link#14 USB lo0
publicipv6 fe80::256:2bff:fe76:b022%igb0 UGHS igb0
fd01:0:0:1::/64 link#18 U vlan01
fd01::1:172:20:20:1 link#14 UHS lo0
fd01::1:172:20:20:10 link#18 UHS vlan01
fd01:0:0:2::/64 link#19 U vlan02
fd01::2:172:20:22:1 link#14 UHS lo0
fd01:0:0:3::/64 link#20 U vlan03
fd01::3:172:20:24:1 link#14 UHS lo0
fd01:0:0:4::/64 link#5 U igb4
fd01::4:172:20:19:1 link#14 UHS lo0
fd01:0:0:8::/64 fd01::a:172:20:0:0 UGS ixl2
fd01::a:10:10:0:0/126 fd01::a:172:20:0:0 UGS ixl2
fd01::a:172:20:0:0/127 link#12 U ixl2
fd01::a:172:20:0:1 link#14 UHS lo0
fd01::a:172:20:21:0 link#14 UHS lo0
fd01::a:172:20:21:0/127 link#13 U ixl3
fd01::a:172:20:21:4/127 link#2 U igb1
fd01::a:172:20:21:5 link#14 UHS lo0
fd01::a:172:20:25:0 link#14 UHS lo0
fd01::a:172:20:25:0/127 link#3 U igb2
fd01::a:172:20:253:2/127 link#22 U wg1
fd01::a:172:20:253:3 link#14 UHS lo0
fd01:0:0:f::/64 link#4 U igb3
fd01:0:0:f::1 link#14 UHS lo0
fe80::%igb0/64 link#1 U igb0
fe80::a236:9fff:fe89:60e7%lo0 link#14 UHS lo0
fe80::%igb1/64 link#2 U igb1
fe80::7ec2:55ff:fe2e:2c71%lo0 link#14 UHS lo0
fe80::%igb2/64 link#3 U igb2
fe80::7ec2:55ff:fe2e:2c72%lo0 link#14 UHS lo0
fe80::%igb3/64 link#4 U igb3
fe80::7ec2:55ff:fe2e:2c73%lo0 link#14 UHS lo0
fe80::%igb4/64 link#5 U igb4
fe80::7ec2:55ff:fe2e:2c74%lo0 link#14 UHS lo0
fe80::%igb6/64 link#7 U igb6
fe80::7ec2:55ff:fe2e:2c76%lo0 link#14 UHS lo0
fe80::%ixl2/64 link#12 U ixl2
fe80::7ec2:55ff:fe25:88%lo0 link#14 UHS lo0
fe80::%ixl3/64 link#13 U ixl3
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%lo0/64 link#14 U lo0
fe80::1%lo0 link#14 UHS lo0
fe80::%vlan01/64 link#18 U vlan01
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan02/64 link#19 U vlan02
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan03/64 link#20 U vlan03
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan04/64 link#21 U vlan04
fe80::7ec2:55ff:fe2e:2c76%lo0 link#14 UHS lo0
Interfaces:
nat64: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4080000<LINKSTATE,MEXTPG>
inet 172.20.0.1 --> 192.168.255.1 netmask 0xffffffff
groups: tun tayga
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 85915
drivername: tun0
vlan01: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: EdgeNet (opt7)
options=4000000<MEXTPG>
ether 7c:c2:55:25:00:89
inet 172.20.20.1 netmask 0xffffff00 broadcast 172.20.20.255
inet6 fd01::1:172:20:20:1 prefixlen 64
inet6 fe80::7ec2:55ff:fe25:89%vlan01 prefixlen 64 scopeid 0x12
groups: vlan
vlan: 120 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
drivername: vlan0
ixl3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: DMZSRV (opt2)
options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
ether 7c:c2:55:25:00:89
inet6 fd01::a:172:20:21:0 prefixlen 127
inet6 fe80::7ec2:55ff:fe25:89%ixl3 prefixlen 64 scopeid 0xd
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
drivername: ixl3
igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
ether a0:36:9f:89:60:e7
hwaddr 7c:c2:55:2e:2c:70
inet publicipv4 netmask 0xffffff00 broadcast 255.255.255.255
inet6 fe80::a236:9fff:fe89:60e7%igb0 prefixlen 64 scopeid 0x1
inet6 publicipv6 prefixlen 128 pltime 86400 vltime 86400
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
drivername: igb0
Client test : (
$ nc -6 -w 1 -vz grc.com 443
Ncat: Version 7.95 ( https://nmap.org/ncat )
Ncat: TIMEOUT.
I have DNS64 setup correctly, clients are resolving the virtual IPv6 address, however they have been timing out and falling back to ipv4 (when they have an IPv4 address).
Tayga config:
IPv4 Address 192.168.255.1
IPv4 NAT64 Interface Address 172.20.0.1
IPv6 Address _blank_
IPv6 NAT64 Interface Address fd01::a:172:20:0:1
IPv6 Prefix 64:ff9b::/96
IPv4 Pool 192.168.255.0/24
"Only used for ICMP."
This phrase in the tayga config is doing *a lot* of heavy lifting. (It makes it seem like its essentially useless if you dont care about icmp, which for a pure netcat TCP test, I dont)
NAT Oubound rule:
Interface Source Source Port Destination Destination Port NAT Address NAT Port Static Port Description
WAN 192.168.255.0/24 * * * Interface address * NO NAT64 Tayga Outbound NAT
Tayga Interface rule (allow all):
Pass IN Tayga IPv4+IPv6 * * * * * *
Looks like I've hit all the points in the setup wiki https://docs.opnsense.org/manual/how-tos/tayga.html
And from my troubleshooting below, it seems like the outbound nat, firewall rule, and tayga itself are all operating properly.
I think I have it narrowed down to the internal IPv6 return traffic being dropped by the kernel.
My tcpdumps are showing:
WAN Interface
[root@EFW ~]# tcpdump -vvvniigb0 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:01:18.926683 IP (tos 0x0, ttl 88, id 25707, offset 0, flags [none], proto TCP (6), length 52)
publicIPv4.42260 > 4.79.142.200.443: Flags [ S ], cksum 0x35d7 (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.960649 IP (tos 0x0, ttl 121, id 16200, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0xf1e9 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965648 IP (tos 0x0, ttl 121, id 17602, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0xf1e9 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:27.971536 IP (tos 0x0, ttl 121, id 18655, offset 0, flags [DF], proto TCP (6), length 48)
4.79.142.200.443 > publicIPv4.42260: Flags [S.], cksum 0x25f9 (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:39.967678 IP (tos 0x0, ttl 121, id 20842, offset 0, flags [DF], proto TCP (6), length 40)
4.79.142.200.443 > publicIPv4.42260: Flags [R], cksum 0x52c9 (correct), seq 1881125065, win 0, length 0
5 packets captured
4058 packets received by filter
0 packets dropped by kernel
Tayga Interface
[root@EFW ~]# tcpdump -vvvninat64 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on nat64, link-type NULL (BSD loopback), snapshot length 262144 bytes
18:01:18.926619 IP6 (flowlabel 0x6a58e, hlim 90, next-header TCP (6) payload length: 32) fd01::1:172:20:20:10.39346 > 64:ff9b::44f:8ec8.443: Flags [ S ], cksum 0xff4d (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.926638 IP (tos 0x0, ttl 89, id 25707, offset 0, flags [none], proto TCP (6), length 52)
192.168.255.195.39346 > 4.79.142.200.443: Flags [ S ], cksum 0x3da6 (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
18:01:18.960684 IP (tos 0x0, ttl 120, id 16200, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0xf9b8 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:18.960718 IP6 (hlim 119, next-header TCP (6) payload length: 32) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xbb60 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965711 IP (tos 0x0, ttl 120, id 17602, offset 0, flags [DF], proto TCP (6), length 52)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0xf9b8 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:21.965742 IP6 (hlim 119, next-header TCP (6) payload length: 32) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xbb60 (correct), seq 1881125064, ack 1104216989, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:01:27.971559 IP (tos 0x0, ttl 120, id 18655, offset 0, flags [DF], proto TCP (6), length 48)
4.79.142.200.443 > 192.168.255.195.39346: Flags [S.], cksum 0x2dc8 (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:27.971590 IP6 (hlim 119, next-header TCP (6) payload length: 28) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [S.], cksum 0xef6f (correct), seq 1881125064, ack 1104216989, win 65535, options [mss 1460,nop,nop,sackOK], length 0
18:01:39.967745 IP (tos 0x0, ttl 120, id 20842, offset 0, flags [DF], proto TCP (6), length 40)
4.79.142.200.443 > 192.168.255.195.39346: Flags [R], cksum 0x5a98 (correct), seq 1881125065, win 0, length 0
18:01:39.967789 IP6 (hlim 119, next-header TCP (6) payload length: 20) 64:ff9b::44f:8ec8.443 > fd01::1:172:20:20:10.39346: Flags [R], cksum 0x1c40 (correct), seq 1881125065, win 0, length 0
10 packets captured
10 packets received by filter
0 packets dropped by kernel
Internal Interface
[root@EFW ~]# tcpdump -vvvnivlan01 host 64:ff9b::44f:8ec8 or host 4.79.142.200
tcpdump: listening on vlan01, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:01:18.926584 IP6 (flowlabel 0x6a58e, hlim 91, next-header TCP (6) payload length: 32) fd01::1:172:20:20:10.39346 > 64:ff9b::44f:8ec8.443: Flags [ S ], cksum 0xff4d (correct), seq 1104216988, win 64800, options [mss 1440,nop,nop,sackOK,nop,wscale 7], length 0
1 packet captured
2694 packets received by filter
0 packets dropped by kernel
I think this one counter `failures of source address selection` is the symptom, as it tends to increase ~60seconds after each test
[root@EFW ~]# netstat -s -p ip6
ip6:
34443359 total packets received
0 with size smaller than minimum
0 with data size < data length
0 with bad options
276 with incorrect version number
0 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 fragments that exceeded limit
0 atomic fragments
0 packets reassembled ok
1864797 packets for this host
31895889 packets forwarded
0 packets not forwardable
0 redirects sent
3043173 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
2 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
4 packets that violated scope rules
62 multicast packets which we don't join
Input histogram:
hop by hop: 699
TCP: 32210783
UDP: 1981356
ICMP6: 250233
PIM: 12
Mbuf statistics:
16835632 one mbuf
two or more mbuf:
lo0= 2124
wg1= 377865
17227738 one ext mbuf
0 two or more ext mbuf
0 packets whose headers are not contiguous
0 tunneling packets that can't find gif
0 packets discarded because of too many headers
2648 failures of source address selection
source addresses on an outgoing I/F
53783 link-locals
77070 globals
source addresses on a non-outgoing I/F
82 globals
2648 addresses scope=0xf
source addresses of same scope
53780 link-locals
77152 globals
source addresses of a different scope
3 link-locals
Source addresses selection rule applied:
130935 first candidate
15095 same address
53708 appropriate scope
48685 outgoing interface
82 matching label
42921 longest match
Fw info: (OPNsense 26.1.8_5-amd64)
routes:
[root@EFW ~]# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default publicipv4gw UGS igb0
1.1.1.1 publicipv4gw UGHS igb0
10.10.0.0/16 172.20.0.2 UGS ixl2
publicipv4block/24 link#1 U igb0
publicipv4 link#14 UHS lo0
127.0.0.1 link#14 UH lo0
172.20.0.0/30 link#12 U ixl2
172.20.0.1 link#14 UHS lo0
172.20.19.0/29 link#5 U igb4
172.20.19.1 link#14 UHS lo0
172.20.20.0/24 link#18 U vlan01
172.20.20.1 link#14 UHS lo0
172.20.21.4/30 link#2 U igb1
172.20.21.5 link#14 UHS lo0
172.20.22.0/24 link#19 U vlan02
172.20.22.1 link#14 UHS lo0
172.20.24.0/24 link#20 U vlan03
172.20.24.1 link#14 UHS lo0
172.20.253.0/30 link#22 U wg1
172.20.253.1 link#14 UHS lo0
192.168.12.0/24 link#4 U igb3
192.168.12.1 link#14 UHS lo0
192.168.255.0/24 link#24 US nat64
192.168.255.1 link#24 UH nat64
Internet6:
Destination Gateway Flags Netif Expire
default fe80::256:2bff:fe76:b022%igb0 UG igb0
::1 link#14 UHS lo0
64:ff9b::/96 link#24 US nat64
publicipv6 link#14 UHS lo0
publicipv6prefix::/60 link#14 USB lo0
publicipv6 fe80::256:2bff:fe76:b022%igb0 UGHS igb0
fd01:0:0:1::/64 link#18 U vlan01
fd01::1:172:20:20:1 link#14 UHS lo0
fd01::1:172:20:20:10 link#18 UHS vlan01
fd01:0:0:2::/64 link#19 U vlan02
fd01::2:172:20:22:1 link#14 UHS lo0
fd01:0:0:3::/64 link#20 U vlan03
fd01::3:172:20:24:1 link#14 UHS lo0
fd01:0:0:4::/64 link#5 U igb4
fd01::4:172:20:19:1 link#14 UHS lo0
fd01:0:0:8::/64 fd01::a:172:20:0:0 UGS ixl2
fd01::a:10:10:0:0/126 fd01::a:172:20:0:0 UGS ixl2
fd01::a:172:20:0:0/127 link#12 U ixl2
fd01::a:172:20:0:1 link#14 UHS lo0
fd01::a:172:20:21:0 link#14 UHS lo0
fd01::a:172:20:21:0/127 link#13 U ixl3
fd01::a:172:20:21:4/127 link#2 U igb1
fd01::a:172:20:21:5 link#14 UHS lo0
fd01::a:172:20:25:0 link#14 UHS lo0
fd01::a:172:20:25:0/127 link#3 U igb2
fd01::a:172:20:253:2/127 link#22 U wg1
fd01::a:172:20:253:3 link#14 UHS lo0
fd01:0:0:f::/64 link#4 U igb3
fd01:0:0:f::1 link#14 UHS lo0
fe80::%igb0/64 link#1 U igb0
fe80::a236:9fff:fe89:60e7%lo0 link#14 UHS lo0
fe80::%igb1/64 link#2 U igb1
fe80::7ec2:55ff:fe2e:2c71%lo0 link#14 UHS lo0
fe80::%igb2/64 link#3 U igb2
fe80::7ec2:55ff:fe2e:2c72%lo0 link#14 UHS lo0
fe80::%igb3/64 link#4 U igb3
fe80::7ec2:55ff:fe2e:2c73%lo0 link#14 UHS lo0
fe80::%igb4/64 link#5 U igb4
fe80::7ec2:55ff:fe2e:2c74%lo0 link#14 UHS lo0
fe80::%igb6/64 link#7 U igb6
fe80::7ec2:55ff:fe2e:2c76%lo0 link#14 UHS lo0
fe80::%ixl2/64 link#12 U ixl2
fe80::7ec2:55ff:fe25:88%lo0 link#14 UHS lo0
fe80::%ixl3/64 link#13 U ixl3
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%lo0/64 link#14 U lo0
fe80::1%lo0 link#14 UHS lo0
fe80::%vlan01/64 link#18 U vlan01
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan02/64 link#19 U vlan02
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan03/64 link#20 U vlan03
fe80::7ec2:55ff:fe25:89%lo0 link#14 UHS lo0
fe80::%vlan04/64 link#21 U vlan04
fe80::7ec2:55ff:fe2e:2c76%lo0 link#14 UHS lo0
Interfaces:
nat64: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4080000<LINKSTATE,MEXTPG>
inet 172.20.0.1 --> 192.168.255.1 netmask 0xffffffff
groups: tun tayga
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 85915
drivername: tun0
vlan01: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: EdgeNet (opt7)
options=4000000<MEXTPG>
ether 7c:c2:55:25:00:89
inet 172.20.20.1 netmask 0xffffff00 broadcast 172.20.20.255
inet6 fd01::1:172:20:20:1 prefixlen 64
inet6 fe80::7ec2:55ff:fe25:89%vlan01 prefixlen 64 scopeid 0x12
groups: vlan
vlan: 120 vlanproto: 802.1q vlanpcp: 0 parent interface: ixl3
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
drivername: vlan0
ixl3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: DMZSRV (opt2)
options=48500b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
ether 7c:c2:55:25:00:89
inet6 fd01::a:172:20:21:0 prefixlen 127
inet6 fe80::7ec2:55ff:fe25:89%ixl3 prefixlen 64 scopeid 0xd
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
nd6 options=121<PERFORMNUD,AUTO_LINKLOCAL,NO_DAD>
drivername: ixl3
igb0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=48520b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,HWSTATS,MEXTPG>
ether a0:36:9f:89:60:e7
hwaddr 7c:c2:55:2e:2c:70
inet publicipv4 netmask 0xffffff00 broadcast 255.255.255.255
inet6 fe80::a236:9fff:fe89:60e7%igb0 prefixlen 64 scopeid 0x1
inet6 publicipv6 prefixlen 128 pltime 86400 vltime 86400
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
drivername: igb0
Client test : (
$ nc -6 -w 1 -vz grc.com 443
Ncat: Version 7.95 ( https://nmap.org/ncat )
Ncat: TIMEOUT.
