flickering Virtual IPs when IPv6 addresses are used

[jli]

We've been running an OPNsense firewall for some time. It's a VM running on a proxmox host. Everything was running fine, but since the update to 25.1.x, we've been experiencing some unpleasant problems with virtual IPs on the WAN port.

We have defined multiple virtual IPs on the WAN port because several systems behind the firewall use official IPs to the outside world. Everything's fine if the virtual IP is an IPv4 address. BUT, if we assign additional official IPv6 addresses as virtual IPs to the WAN port, we've (recently) started having problems.

At first, the IPv6 address is reachable for a few minutes. Suddenly, the connection drops. The address is then no longer reachable via ping6. After a few minutes, the IPv6 address is reachable again, only to disappear again for a while.

While the connection is intermittent, other IPv4 addresses also implemented via virtual IP are constantly reachable.

Does anyone have any idea what might be causing this? We're fairly certain this issue didn't occur with OPNsense 24.7.

[franco]

Which version are you running now, 25.1 or 25.1.3?


Cheers,
Franco

[jli]

* It was OPNsense 25.1.2-amd64 till 2 hours ago.

* I've just updated to OPNsense 25.1.3-amd64.
-> Problems still remain

[franco]

Anything in dmesg on the box? It sounds like addresses are bouncing between machines.


Cheers,
Franco

[jli]

There is only one OPNsense in our network. Do there is noch failover or second machine the Virtual IPs could be transferred to. What I forgot to mention is: the IPv6-Address of the WAN-Port is stable. It's a problem vor the Virtual IPs only.

there are messages in dmesg

Code Select Expand

cannot forward from fe80:1::9a03:9bff:fee3:a800 to fd02:993:9ca6:20::116 nxt 58 received on vtnet0
cannot forward from fe80:1::9a03:9bff:fee3:a800 to fd02:993:9ca6:20::116 nxt 58 received on vtnet0
cannot forward src fe80:1::9a03:9bff:fee3:a800, dst fd02:993:9ca6:10::117, nxt 58, rcvif vtnet0, outif vtnet1
cannot forward from fe80:1::9a03:9bff:fee3:a800 to fd02:993:9ca6:20::116 nxt 58 received on vtnet0
cannot forward src fe80:1::9a03:9bff:fee3:a800, dst fd02:993:9ca6:10::117, nxt 58, rcvif vtnet0, outif vtnet1
cannot forward from fe80:1::9a03:9bff:fee3:a800 to fd02:993:9ca6:20::116 nxt 58 received on vtnet0
cannot forward src fe80:1::9a03:9bff:fee3:a800, dst fd02:993:9ca6:10::117, nxt 58, rcvif vtnet0, outif vtnet1
cannot forward from fe80:1::9a03:9bff:fee3:a800 to fd02:993:9ca6:20::116 nxt 58 received on vtnet0
cannot forward from fe80:1::9a03:9bff:fee3:a800 to fd02:993:9ca6:20::116 nxt 58 received on vtnet0
cannot forward from fe80:1::9a03:9bff:fee3:a800 to fd02:993:9ca6:20::116 nxt 58 received on vtnet0
cannot forward src fe80:1::9a03:9bff:fee3:a800, dst fd02:993:9ca6:20::116, nxt 58, rcvif vtnet0, outif vtnet2
cannot forward src fe80:1::9a03:9bff:fee3:a800, dst fd02:993:9ca6:20::116, nxt 58, rcvif vtnet0, outif vtnet2
cannot forward src fe80:1::9a03:9bff:fee3:a800, dst fd02:993:9ca6:20::116, nxt 58, rcvif vtnet0, outif vtnet2
fd02:993:9ca6:20::116 is the local IPv6 Address of a VM behind the firewall. There are some NPTv6 settings in place to NAT this IPv6 to a valid global IPv6

[jli]

To be honest, I'm not sure why there errors show up.

[Patrick M. Hausen]

The source fe80:1::9a03:9bff:fee3:a800 is a link local address on vtnet0. Link local addresses cannot communicate across interfaces.

[jli]

Yes, the fe80:1:: is some kind of link lokal router address.

But, I've no idea why this shows up and how this is related to the availability problem?

To summarise:
* fd02:993:9ca6:20::116 is the unique local adress of one of the VMs
* xxxx:xxx:2000:117::117/64 is the Virtual IP that the above VM is available from the outside via NPTv6

it does work for some minutes. than stops for some time. Works again... stops again...

[meyergru]

Just an idea - are you aware that IPv6 neighbor discovery does not work on Proxmox VMs unless you have multicast snooping disabled on your bridge interface?

This is in my tutorial in the network section...

[jli]

meyergru, thank you very much for the tip and the idea. We modified the bridge accordingly on the proxmox host.

Unfortunately, it didn't solve the problem. The Alias IPs are still not consistently available.

[meyergru]

Oh, BTW: formally, fe80:1::9a03:9bff:fee3:a800 is not a link-local address, but IDK if that matters.

[franco]

As far as I know the embedded scope "1" cannot appear on the wire, but it also should not be added in the configuration. It's the duty of the OS to handle the scoping.


Cheers,
Franco