PING/Unreachable Virtual IPv6 Addresses on WAN Interface

[zcdigi]

Hello.

Hopefully the answer to this is that I'm just doing something silly.

I'm testing OPNsense in a VM.

I've set up the WAN interface with 2 virtual IPv6 addresses. I'm able to access these from within the router (PING, DNS, etc.), as specified in the firewall rules.

But when outside the network, I'm only able to access the primary IPv6 address. PING shows the other addresses as unreachable.

I've tried setting up the firewall with "Destination allow: WAN address", "Destination allow: WAN net", "Destination allow: this firewall", "Destination allow: single host (with IP)" as well as "Destination allow: network (with network subnet)".

What am I doing wrong?

Thank you in advance.

Sam

[Monviech (Cedrik)]

Are the virtual ipv6 addresses /128 or /64? They should be /64 if the parent interface is also /64.

[Maurice]

Is this actually a firewall issue? What does the firewall log say? Do you see these pings getting blocked?

Cheers
Maurice

[zcdigi]

@Moneviech

Yes they are all /64. across the board.

@maurice

I don't actually know how to do that, view logs, let me figure it out. I am able to ping the source machine IPs from the OPNsense router though.

[Monviech (Cedrik)]

Can you tell me how the ipv6 network is connected to the VM?
Is the WAN interface static IPv6, or pppoe with prefix delegation, or dhcpv6 or slaac? I just want to rule out its a routing issue if you can ping it internally but not externally.

[zcdigi]

@Monviech

The OPNsense router (OR) is running in a VM on a Linux machine without a firewall, behind the ISP router. The IPv6 prefix is assigned by the ISP, all IPv6 addresses on the network use the same prefix/netmask. The WAN/LAN interfaces are in bridged mode. The primary IPv6 address on the WAN IF on the OR is a static address, and this can be pinged and is reachable from the WAN interface. The gateway is defined the same as the IPv6 address assigned to the router (checked using netstat -rn6 from within the OR). One of virtual IPv6 addresses I've assigned to the WAN interface is the autoconfig SLAAC address.

I've also assigned multiple IPv6 addresses to the LAN interface. All IPs are pingable from the host machine.

I've also turned off blocking traffic from private network addresses on the OR. I've also added a firewall rule to allow all ICMP traffic to the [ipv6 prefix]/64 network on the WAN.

I'll post more with firewall logs once I understand how to do this.

Thanks for your help.

Code: [Select]

# --- TO LAN ------------------------
❯ ping -c 1 <IPv6 Prefix>::ac10:1111
  ping -c 1 <IPv6 Prefix>::ac10:1010
  ping -c 1 <IPv6 Prefix>:a00:27ff:fee5:7b33

PING <IPv6 Prefix>::ac10:1111(<IPv6 Prefix>::ac10:1111) 56 data bytes
64 bytes from <IPv6 Prefix>::ac10:1111: icmp_seq=1 ttl=64 time=0.628 ms

--- <IPv6 Prefix>::ac10:1111 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.628/0.628/0.628/0.000 ms
PING <IPv6 Prefix>::ac10:1010(<IPv6 Prefix>::ac10:1010) 56 data bytes
64 bytes from <IPv6 Prefix>::ac10:1010: icmp_seq=1 ttl=64 time=1.09 ms

--- <IPv6 Prefix>::ac10:1010 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.092/1.092/1.092/0.000 ms
PING <IPv6 Prefix>:a00:27ff:fee5:7b33(<IPv6 Prefix>:a00:27ff:fee5:7b33) 56 data bytes
64 bytes from <IPv6 Prefix>:a00:27ff:fee5:7b33: icmp_seq=1 ttl=64 time=1.09 ms

--- <IPv6 Prefix>:a00:27ff:fee5:7b33 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.090/1.090/1.090/0.000 ms

# --- TO WAN ------------------------
❯ ping -c 1 -6 <IPv6 Prefix>::c0a8:1111
  ping -c 1 -6 <IPv6 Prefix>::c0a8:10
  ping -c 1 -6 <IPv6 Prefix>:a00:27ff:fe7a:5f94

PING <IPv6 Prefix>::c0a8:1111(<IPv6 Prefix>::c0a8:1111) 56 data bytes
64 bytes from <IPv6 Prefix>::c0a8:1111: icmp_seq=1 ttl=64 time=0.337 ms

--- <IPv6 Prefix>::c0a8:1111 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.337/0.337/0.337/0.000 ms
PING <IPv6 Prefix>::c0a8:10(<IPv6 Prefix>::c0a8:10) 56 data bytes
From <IPv6 Prefix>:10ba:7ce7:2315:720 icmp_seq=1 Destination unreachable: Address unreachable

--- <IPv6 Prefix>::c0a8:10 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING <IPv6 Prefix>:a00:27ff:fe7a:5f94(<IPv6 Prefix>:a00:27ff:fe7a:5f94) 56 data bytes
From <IPv6 Prefix>:10ba:7ce7:2315:720 icmp_seq=1 Destination unreachable: Address unreachable

--- <IPv6 Prefix>:a00:27ff:fe7a:5f94 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms


[Monviech (Cedrik)]

Can you check the NDP (Neighbor Discovery Protocol) Table on your ISP router? It's like ARP for IPv6. It would show if the ISP router knows that the additional IPv6 addresses exist on the MAC address of the OPNsense.

To get an idea how it looks like, you can see it in the OPNsense too:
Interfaces: Diagnostics: NDP Table

All of your other VMs that can ping the virtual IP addresses of the OPNsense should also have the virtual IPs + MAC address in their NDP Tables. I think on linux its "ip -6 neighbour"

If the NDP Table is correct everywhere, it has to be probably a firewall issue like @Maurice suggested at first.

[zcdigi]

How do I turn off the firewall? That would be quickest way to check if it's a firewall issue?

Thanks all.

@Moneviech - I checked NDP on host and router. Everything seems to be OK.

ADDED NOTES - STRANGENESS! --

On the firewall, I have (paraphrasing) "allow all ICMP to all <IPv6 Prefix>::/64 network".

OK so I switched LAN/WAN IPv6 addressing from static to SLAAC, deleted then added all the virtual IPv6 addresses again, to both LAN/WAN, including ones with the SLAAC related suffix. Then reboot. All addresses show correctly when I run "ifconfig".

And I'm still getting the exact same 2 IP addresses being unreachable! Also I'm unable to use as source IP.
There's no loss when I ping internally to these 2 IP addresses from within the router.

Not what I expected...

This is what I'm getting now -

Code: [Select]

## -- on router --
# ifconfig em[0|1]
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN (lan)
        options=4800008<VLAN_MTU,NOMAP>
        ether 08:00:27:7a:5f:94
        inet 172.16.16.16 netmask 0xfffff000 broadcast 172.16.31.255
        inet6 fe80::a00:27ff:fe7a:5f94%em0 prefixlen 64 scopeid 0x1
        inet6 <IPv6 Prefix>::ac10:1111 prefixlen 64
        inet6 <IPv6 Prefix>::ac10:1010 prefixlen 64
        inet6 <IPv6 Prefix>:a00:27ff:fe7a:5f94 prefixlen 64
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
em1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN (wan)
        options=4800008<VLAN_MTU,NOMAP>
        ether 08:00:27:e5:7b:33
        inet 192.168.0.16 netmask 0xfffffc00 broadcast 192.168.3.255
        inet6 fe80::a00:27ff:fee5:7b33%em1 prefixlen 64 scopeid 0x2
        inet6 <IPv6 Prefix>::c0a8:1111 prefixlen 64
        inet6 <IPv6 Prefix>::c0a8:10 prefixlen 64
        inet6 <IPv6 Prefix>:a00:27ff:fee5:7b33 prefixlen 64
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

## -- from host machine to router LAN address --
ping -6 -c 1 <IPv6 Prefix>:a00:27ff:fe7a:5f94 ## OK
ping -6 -c 1 <IPv6 Prefix>::ac10:1111 ## OK
ping -6 -c 1 <IPv6 Prefix>::ac10:1010 ## OK
## -- from host machine to router WAN address --
ping -6 -c 1 <IPv6 Prefix>:a00:27ff:fee5:7b33 ## FAIL
ping -6 -c 1 <IPv6 Prefix>::c0a8:1111 ## OK
ping -6 -c 1 <IPv6 Prefix>::c0a8:10 ## FAIL

## -- from router, LAN address --
ping6 -c 1 -S <IPv6 Prefix>:a00:27ff:fe7a:5f94 <hostip> ## OK
ping6 -c 1 -S <IPv6 Prefix>::ac10:1111 <hostip> ## OK
ping6 -c 1 -S <IPv6 Prefix>::ac10:1010 <hostip> ## OK
## -- from router, WAN address --
ping6 -c 1 -S <IPv6 Prefix>:a00:27ff:fee5:7b33 <hostip> ## FAIL
ping6 -c 1 -S <IPv6 Prefix>::c0a8:1111 <hostip> ## OK
ping6 -c 1 -S <IPv6 Prefix>::c0a8:10 <hostip> ## FAIL


[Maurice]

I'm not sure I follow you, but regarding the ifconfig: Is <IPv6 Prefix> the same for LAN and WAN?

[zcdigi]

@maurice

Yes, IPv6 Prefix is the same.


UPDATE --
I already had a firewall rule "allow all ICMP to all <IPv6 Prefix>::/64 network". I added an additional rule "allow from <IPv6 Prefix>::/64 network to all". Now all addresses are reachable and reachable/pingable!

Doesn't make sense to me, but I'll take it for now.

[Maurice]

Don't use the same /64 on WAN and LAN! That's a big no-no and will cause all kinds of issues as you continue. It's the first thing you should fix.

[Patrick M. Hausen]

If you only have a single /64 it is entirely possible to use a single address of that prefix on WAN with a /128 prefix length and the rest of the prefix on LAN. We do this outside of the OPßnsense context with our FreeBSD hosts located at Hetzner.

At least with the FreeBSD stack this is the one exception to the rule that one L3 prefix can be configured on one interface/L2 broadcast domain only. You can pick individual addresses as /32 or /128 and assign them elsewhere. Of course that essentially eliminates the broadcast characteristic of that interface. So you would e.g. use a link local address with scope as your WAN default gateway.

HTH,
Patrick

[zcdigi]

Hi @Patrick @Maurice

Thank you for your input.

I'll look into how I can implement this, re. subnets on IPv6.

If you have any resources you can point me to, that would be great.

I was mainly concerned about the broadcast address if I use subnets other than /64.

@all thanks very much for your help. Still learning.

[Maurice]

@Patrick Absolutely, but I wouldn't necessarily call this an "exception to the rule that one L3 prefix can be configured on one interface/L2 broadcast domain only". You're not configuring the same prefix on multiple interfaces after all. You can also configure e.g. 2001:db8:1:1::/64 on one interface and 2001:db8:1:1:1::/80 on another one.

Cheers
Maurice

[Patrick M. Hausen]

@Maurice yeah ... more specifics. You win 

But, hey, on a PTP interface I can reuse even the same IP address as on my LAN as a /128 ...

@zcdigi concerning "subnets" - you don't want that. About everything that is taken for granted with IPv6 will break if you put anything but a /64 on an Ethernet broadcast medium.