Messages

Thanks! I will check both export options, as I ended up with disabling netflow - vacuum caused also CARP flapping.

Writing above post triggered better thinking ;-). The culprit was selection of VPNs' interfaces in netflow settings. Once they've been removed, all went back to normal...

So it seems all started with upgrade to 26.1 and at that time (January) issues with Neighbours Discovery. That was sorted out by the next upgrades/fixes but I started having similar issues with netflow/rrd. Once they are enabled, they cause high I/O and CPU demand. I tried, repair and reset netflow/rrd data plus manual removal of content of /various/netflow/ folder (with stopped neighbours discovery and netflow). Nothing helps; currently system is up to date: 26.1.6.
How to fix it?

So, for those who may experience similar issues:

• WG (only in HA CARP mode) requires additional outbound rule for LAN interfaces and source as WG net
• Very unexpected reason of duplicated data was my set up of CARP VIPs in unicast mode. After removing peers (back to multicast CARP), all of sudden there's no more duplicated data flow and pings plus no webgui/ssh flapping... I wonder why?

One more thing to add is when I reach my LAN over VPN (either Wireguard or OpenVPN) I can't communicate with backup instance (its physical interface addresses) at all while FW rules allow them to send requests to any hosts...

With your assistance in previous topics, I got HA in working condition, but...

To describe my setup:

• 2x Opnsense instances in high availability mode with carp vip interfaces on single pve host. I know it's not full HA but I want software HA and also simply to test it.
• VMs are connected through 3 bridges: 1 on WAN side, the other on LAN side (and further trunk physical link to switch) and pfsync bridge.
• IGMP snooping, storm control are disabled in (UniFi) switches.

In order to change above configuration and (trying to) test my issue, I created additional LAN bridge for backup instance and instead of having them (2x opnsense) connected over single linux bridge - within proxmox, I connected them over physical switch.
This of course requires second downlink:

• master/regular LAN bridge would remain connected as it is now
• backup/new LAN bridge is connected to switch via additional downlink

But problem I'm facing is duplicated communication/data flow to and from both VMs; both instances have same looking graphs in proxmox webgui - network flow and also cpu. Despite they don't change their master/backup status (no flapping at carp status) I have something similar to split brain situation, for example if I communicate with opnsense webgui or ssh on carp vip interface, reply comes either from one of those two and toggles every few seconds. If I ping them, reply is duplicated ("DUP!"). Communication to other hosts and WAN is ok.  I have already set Mac filter to "no" in proxmox VM's firewall options (pve firewall is disabled). I tried ovs and Linux bridges with same results.

To me, it is something related to MAC and network switches; is it possible to set it up correctly?

An update: out of blue (almost) ipv6 started working! My guess is that is because of "routes" - I had them configured in my previous setup. I deleted them from RA when preparing HA setup, but maybe my laptop had cached them (?)... Anyway, since couple of hours ago it did start working and continue doing so...

Check for these:
- If you set a source address for the RAs, but "cat /var/etc/radvd.conf" does not contain it.
- If you set a source address for the RAs, and packet capture that the source address of the RAs (Source link layer option) is not the source address you set.

radvd.conf contains source address:

Code Select Expand

[color=#000000][size=1][font=Menlo][/font][/size][/color]
interface vlan14 {
    AdvSendAdvert on;
    MinRtrAdvInterval 200;
    MaxRtrAdvInterval 600;
    AdvLinkMTU 1500;
    AdvDefaultPreference high;
    AdvRASrcAddress {        fe80::14;
    };
    AdvSourceLLAddress off;
    RemoveAdvOnExit off;
    prefix XXXXXXXXd:4::/64 {        DeprecatePrefix off;
        AdvOnLink on;
        AdvAutonomous on;
    };
    RDNSS XXXXXXXXXd:1::4 {    };
    DNSSL x.xx {    };
};

and tcpdump of RA:

Code Select Expand

tcpdump -i vlan14 -vv -n icmp6 and 'ip6[40] == 134'

tcpdump: listening on vlan14, link-type EN10MB (Ethernet), snapshot length 262144 bytes11:17:40.481739 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 112) fe80::14 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 112
hop limit 64, Flags [other stateful], pref high, router lifetime 1800s, reachable time 0ms, retrans timer 0ms
  prefix info option (3), length 32 (4): XXXXXXXXXd:4::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
    0x0000:  40c0 0001 5180 0000 3840 0000 0000 2001
    0x0010:  0470 604d 0004 0000 0000 0000 0000
  rdnss option (25), length 24 (3):  lifetime 1800s, addr: XXXXXXXd:1::4
    0x0000:  0000 0000 0708 2001 0470 604d 0001 0000
    0x0010:  0000 0000 0004
  dnssl option (31), length 32 (4):  lifetime 1800s, domain(s): x.xx.
    0x0000:  0000 0000 0708 0d6d 6172 737a 616c 6b6f
    0x0010:  7773 6379 0270 6c00 0000 0000 0000
  mtu option (5), length 8 (1):  1500
    0x0000:  0000 0000 05dc

Like I wrote in my first message: 

But as I use tunnelbroker I can't use my ipv4 WAN interface to set up CARP VIP (https://docs.opnsense.org/manual/how-tos/carp.html#setup-virtual-ipv6-global-unicast-address) and I think this should have been my GIF interface...(?) And if I set next hop, either tunnel remote or local address as CARP VIP address, VIP remains as disabled...

This could have been my source of this issue, but I'm not sure how to solve it.

There is no bug here the field exists and you can input the source IP address.

That's what I'd done also (i.e. I typed in: fe80::14) and doesn't work. Once I remove CARP VIPs ipv6 works fine.

My issue may have something to do with tunnelbroker setup, as I don't have native ipv6 provider available... OR I will try also to reconfigure my PVE setup and create additional LAN bridge for backup instance and instead of having them (2x opnsense) connected over single linux bridge - within proxmox, connect them over physical switch?
This of course requires second downlink, so:

• regular bridge would remain connected as it is now
• backup/new bridge will be connected to switch via additional downlink

Please report on Github.

There is issue already created: https://github.com/opnsense/core/issues/9873

Then there's probably a bug. This used to work in CE, too, before we switched to BE.

Do I need to report it or this forum is monitored?

So, it means that ipv6 won't work in HA setup in community edition...?
The only place I've found to choose interface with carp vip address is gif interface settings.

Radvd config:

Thanks!
Can you confirm, do you use: Services -> Router Advertisements? Because I don't have any dropdown list to choose from...

- the interface configuration of both units
- the CARP VIP configuration on the active/master

So, I adjusted local link addresses - as per your advice to make sure they are different

I couldn't attached screenshots here as limit is only 250 kB (?)
Link to listed below screenshots: https://imgur.com/a/r9RSFma
Master interface: 
Backup interface:
CARP VIP global (I had multicast, same issue, so I tried unicast)
CARP VIP local:


VHID groups are fine, initially I synchronised CARP VIPs, later I changed them to unicast.
So, where's mistake???

I'm not able to set CARP VIPs for link local addresses (https://docs.opnsense.org/manual/how-tos/carp.html#setup-virtual-ipv6-link-local-address) to make IPv6 communication flowing. RA announces current, physical link local ipv6 address as router which is different than CARP VIP. I've tried setting fe80::/64 and fe80::1/64  as CARP VIP without any luck...

But as I use tunnelbroker I can't use my ipv4 WAN interface to set up CARP VIP (https://docs.opnsense.org/manual/how-tos/carp.html#setup-virtual-ipv6-global-unicast-address) and I think this should have been my GIF interface...(?) And if I set next hop, either tunnel remote or local address as CARP VIP address, VIP remains as disabled...

How to set it up properly?