Solved: Devolo Powerline adapters are crap (disappearing IPv6 Multicast)

[ruffy91]

I have two OPNsense behind each other.
The outer OPNsense A has a fixed /56 Subnet.
The inner OPNsense should get a /60 via PD.

But B got a /62 before I manually set up the DHCPv6 Server on A
So now B has a /62 PD from A.
I setup A to give out a /60 PD range from ::10 to ::20
Also I set up B to request a /60 from A via PD (Prefix delegation size 60, send prefix hint yes)

How can I release the /62 from either B or delete the lease on A?
I already change the DUID on B but it keeps getting an /62 and there is no possibility of deleting the leases in DHCPv6 Server

The DHCPv6 lease Table on A looks like this
IPv6 Prefix   IAID   DUID   Start   End   State
2a01:dead:beef:1c42::2000 2a01:dead:beef:1cf0::/62   1   00:03:00:01:b8:be:de:ad:be:ef   2019/08/05 20:56:19   2019/08/05 22:56:19   active

Also B does not use this PD for the LAN interface which is tracking the upstream interface.

[ruffy91]

There is a "Release" button on the Interfaces: Overview page.
But. It does not send a DHCPRELEASE.
I opened a Bug report here: https://github.com/opnsense/core/issues/3628

Maybe someone could try to reproduce the problem and give a feedback if it is working.

You can just do a Packet Capture on your WAN Interface for UDP Port 67 for DHCPv4 or 546 for DHCPv6 and then do a release and renew an look at the capture if you got a DHCPRELEASE message.

[ruffy91]

Further investigations gave me following results:
A = APU with OPNsense
B = VM with VirtIO Net

I have the following configuration
A --vlan tagged 100-- Aruba Switch --vlan tagged 100-- Proxmox (ovswitch vmbr0) ---- vtnet adapter (vlan 100 untagging) <> B

Packets from A to B go through, Packets from B to A (for example ICMPv6) vanish when meeting specific conditions.
I did not yet find the conditions which cause the packets to vanish.
Firewall on Proxmox is completely disabled.
One example is this:
capture on vmbr0 (ovswitch on the host where the B OPNsense VM is running):

Code: [Select]

20:39:56.923028 00:0d:b9:45:9a:f5 (oui Unknown) > 33:33:00:00:00:01 (oui Unknown), ethertype 802.1Q (0x8100), length 74: vlan 100, p 4, ethertype IPv6, (hlim 64, next-header ICMPv6 (58) payload length: 16) fe80::20d:b9ff:fe45:9af5 > ip6-allnodes: [icmp6 sum ok] ICMP6, echo request, seq 0
20:39:56.923133 00:0d:b9:45:9a:f5 (oui Unknown) > 33:33:00:00:00:01 (oui Unknown), ethertype 802.1Q (0x8100), length 74: vlan 100, p 4, ethertype IPv6, (hlim 64, next-header ICMPv6 (58) payload length: 16) fe80::20d:b9ff:fe45:9af5 > ip6-allnodes: [icmp6 sum ok] ICMP6, echo request, seq 0
you see 2 echo requests but there are no replies.
This is what it looks like on Firewall B: (see attached screenshot)

So the packets get lost on vtnet0 adapter.

Is this some bug with VirtIO?
IPv4 is working flawless at the same time.

I would be very glad if someone could help me to further debug the problem.
I am pulling my hair out and ther is not much left 

[ruffy91]

OK, capturing on a ovswitch only shows broadcasts. capturing on the upstream interface to the switch shows the packets.
So the packets get lost somewhere between switch and OPNsense A

[ruffy91]

To continue my monologue:
Packets to ff02::1 (ipv6-allnodes) arrive at A
Packets to ff02::1:2 (ipv6-dhcpservers) do not arrive at A

Blame it to the networking equipment, will try a direct cable next

[ruffy91]

Got it!

Devolo Magic 2 Powerline adapter is shit.
Packets to ff02::1 untagged -> works
Packets to ff02::1:2 untagged -> works
Packets to ff02::1 tagged -> works
Packets to ff02::1:2 tagged -> disappears!

Booh.. cost me 3 evenings.. should have got this earlier