Messages

One more thing to add is when I reach my LAN over VPN (either Wireguard or OpenVPN) I can't communicate with backup instance (its physical interface addresses) at all while FW rules allow them to send requests to any hosts...

With your assistance in previous topics, I got HA in working condition, but...

To describe my setup:

• 2x Opnsense instances in high availability mode with carp vip interfaces on single pve host. I know it's not full HA but I want software HA and also simply to test it.
• VMs are connected through 3 bridges: 1 on WAN side, the other on LAN side (and further trunk physical link to switch) and pfsync bridge.
• IGMP snooping, storm control are disabled in (UniFi) switches.

In order to change above configuration and (trying to) test my issue, I created additional LAN bridge for backup instance and instead of having them (2x opnsense) connected over single linux bridge - within proxmox, I connected them over physical switch.
This of course requires second downlink:

• master/regular LAN bridge would remain connected as it is now
• backup/new LAN bridge is connected to switch via additional downlink

But problem I'm facing is duplicated communication/data flow to and from both VMs; both instances have same looking graphs in proxmox webgui - network flow and also cpu. Despite they don't change their master/backup status (no flapping at carp status) I have something similar to split brain situation, for example if I communicate with opnsense webgui or ssh on carp vip interface, reply comes either from one of those two and toggles every few seconds. If I ping them, reply is duplicated ("DUP!"). Communication to other hosts and WAN is ok.  I have already set Mac filter to "no" in proxmox VM's firewall options (pve firewall is disabled). I tried ovs and Linux bridges with same results.

To me, it is something related to MAC and network switches; is it possible to set it up correctly?

An update: out of blue (almost) ipv6 started working! My guess is that is because of "routes" - I had them configured in my previous setup. I deleted them from RA when preparing HA setup, but maybe my laptop had cached them (?)... Anyway, since couple of hours ago it did start working and continue doing so...

Check for these:
- If you set a source address for the RAs, but "cat /var/etc/radvd.conf" does not contain it.
- If you set a source address for the RAs, and packet capture that the source address of the RAs (Source link layer option) is not the source address you set.

radvd.conf contains source address:

Code Select Expand

[color=#000000][size=1][font=Menlo][/font][/size][/color]
interface vlan14 {
    AdvSendAdvert on;
    MinRtrAdvInterval 200;
    MaxRtrAdvInterval 600;
    AdvLinkMTU 1500;
    AdvDefaultPreference high;
    AdvRASrcAddress {        fe80::14;
    };
    AdvSourceLLAddress off;
    RemoveAdvOnExit off;
    prefix XXXXXXXXd:4::/64 {        DeprecatePrefix off;
        AdvOnLink on;
        AdvAutonomous on;
    };
    RDNSS XXXXXXXXXd:1::4 {    };
    DNSSL x.xx {    };
};

and tcpdump of RA:

Code Select Expand

tcpdump -i vlan14 -vv -n icmp6 and 'ip6[40] == 134'

tcpdump: listening on vlan14, link-type EN10MB (Ethernet), snapshot length 262144 bytes11:17:40.481739 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 112) fe80::14 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 112
hop limit 64, Flags [other stateful], pref high, router lifetime 1800s, reachable time 0ms, retrans timer 0ms
  prefix info option (3), length 32 (4): XXXXXXXXXd:4::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
    0x0000:  40c0 0001 5180 0000 3840 0000 0000 2001
    0x0010:  0470 604d 0004 0000 0000 0000 0000
  rdnss option (25), length 24 (3):  lifetime 1800s, addr: XXXXXXXd:1::4
    0x0000:  0000 0000 0708 2001 0470 604d 0001 0000
    0x0010:  0000 0000 0004
  dnssl option (31), length 32 (4):  lifetime 1800s, domain(s): x.xx.
    0x0000:  0000 0000 0708 0d6d 6172 737a 616c 6b6f
    0x0010:  7773 6379 0270 6c00 0000 0000 0000
  mtu option (5), length 8 (1):  1500
    0x0000:  0000 0000 05dc

Like I wrote in my first message: 

But as I use tunnelbroker I can't use my ipv4 WAN interface to set up CARP VIP (https://docs.opnsense.org/manual/how-tos/carp.html#setup-virtual-ipv6-global-unicast-address) and I think this should have been my GIF interface...(?) And if I set next hop, either tunnel remote or local address as CARP VIP address, VIP remains as disabled...

This could have been my source of this issue, but I'm not sure how to solve it.

There is no bug here the field exists and you can input the source IP address.

That's what I'd done also (i.e. I typed in: fe80::14) and doesn't work. Once I remove CARP VIPs ipv6 works fine.

My issue may have something to do with tunnelbroker setup, as I don't have native ipv6 provider available... OR I will try also to reconfigure my PVE setup and create additional LAN bridge for backup instance and instead of having them (2x opnsense) connected over single linux bridge - within proxmox, connect them over physical switch?
This of course requires second downlink, so:

• regular bridge would remain connected as it is now
• backup/new bridge will be connected to switch via additional downlink

Please report on Github.

There is issue already created: https://github.com/opnsense/core/issues/9873

Then there's probably a bug. This used to work in CE, too, before we switched to BE.

Do I need to report it or this forum is monitored?

So, it means that ipv6 won't work in HA setup in community edition...?
The only place I've found to choose interface with carp vip address is gif interface settings.

Radvd config:

Thanks!
Can you confirm, do you use: Services -> Router Advertisements? Because I don't have any dropdown list to choose from...

- the interface configuration of both units
- the CARP VIP configuration on the active/master

So, I adjusted local link addresses - as per your advice to make sure they are different

I couldn't attached screenshots here as limit is only 250 kB (?)
Link to listed below screenshots: https://imgur.com/a/r9RSFma
Master interface: 
Backup interface:
CARP VIP global (I had multicast, same issue, so I tried unicast)
CARP VIP local:


VHID groups are fine, initially I synchronised CARP VIPs, later I changed them to unicast.
So, where's mistake???

I'm not able to set CARP VIPs for link local addresses (https://docs.opnsense.org/manual/how-tos/carp.html#setup-virtual-ipv6-link-local-address) to make IPv6 communication flowing. RA announces current, physical link local ipv6 address as router which is different than CARP VIP. I've tried setting fe80::/64 and fe80::1/64  as CARP VIP without any luck...

But as I use tunnelbroker I can't use my ipv4 WAN interface to set up CARP VIP (https://docs.opnsense.org/manual/how-tos/carp.html#setup-virtual-ipv6-global-unicast-address) and I think this should have been my GIF interface...(?) And if I set next hop, either tunnel remote or local address as CARP VIP address, VIP remains as disabled...

How to set it up properly?

The documentation is correct. In a default CARP setup where the environment works correctly, you don't need any tunables.

For sure I'm not entitled to say otherwise but...
Demotion/maintenance mode didn't work but after disabling carp or shutting down master node failover worked.
I did check using tcpdump and multicast worked, I had the same results with unicast (setting peer IPs). So I don't know what else could have been wrong to prevent maintenance mode from working (dedicated interfaces connected via also dedicated ovs switch)?
That tunable I found on: https://www.openbsd.org/faq/pf/carp.html ....

And to stop that pfsync storm (caused pfsync states limit reached) which kept happening on backup node after failover master - backup - master, I had to enable on both nodes "Defer pfsync" option.

Ok now works after enabling preemption and group interface failover by adding tunable: 

• Code Select Expand
  net.inet.carp.preempt=1


To either both nodes or master and sync config. BTW, it is not present in docs: https://docs.opnsense.org/manual/how-tos/carp.html#configure-carp

Now, maintenance failover is instant!

And today I noticed that backup node is completely unresponsive. In serial console I could see info: 

• pf states limit reached
• and something about swap running out of space

So I guess that somehow RAM got full (while in master node is/was at around 20% of utilisation). Both VMs have assigned 8 GB of RAM.

The only thing I was trying to do, since yesterday reboot, was to activate carp maintenance mode in master node...

Opnsense version is 26.1.3. When I activate in gui carp maintenance mode, almost nothing happens.
Demotion works:

Code Select Expand

sysctl net.inet.carp.demotion
net.inet.carp.demotion: 240
But advskew is not changed on master (regular setting is 1):

Code Select Expand

ifconfig | grep carp
    carp: MASTER vhid 99 advbase 1 advskew 1
    carp: MASTER vhid 16 advbase 1 advskew 1
    carp: MASTER vhid 166 advbase 1 advskew 1
    carp: MASTER vhid 1 advbase 1 advskew 1
    carp: MASTER vhid 16 advbase 1 advskew 1
    carp: MASTER vhid 17 advbase 1 advskew 1
    carp: MASTER vhid 176 advbase 1 advskew 1
    carp: MASTER vhid 11 advbase 1 advskew 1
    carp: MASTER vhid 116 advbase 1 advskew 1
    carp: MASTER vhid 12 advbase 1 advskew 1
    carp: MASTER vhid 126 advbase 1 advskew 1
    carp: MASTER vhid 13 advbase 1 advskew 1
    carp: MASTER vhid 136 advbase 1 advskew 1
    carp: MASTER vhid 14 advbase 1 advskew 1
    carp: MASTER vhid 146 advbase 1 advskew 1
    carp: MASTER vhid 15 advbase 1 advskew 1

Only when I disable carp or shutdown master, backup is elevated to master mode...
I've seen it it's an old issue/bug. Is there any way to fix it?

EDIT:
Results of tcpdump command for one of interfaces of master node when enabling maintenance mode:

Code Select Expand

tcpdump -ni vlan01 -T carp carp

Code Select Expand

15:23:17.637939 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782732
15:23:18.648116 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782733
15:23:19.658610 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782734
15:23:20.420473 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782735
15:23:22.358801 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782736
15:23:24.298404 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782737
15:23:26.237935 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782738
15:23:28.178964 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782739
15:23:30.118701 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782740
15:23:32.058947 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782741
15:23:33.998853 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782742
15:23:35.938963 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782743
15:23:37.878589 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782744
15:23:39.818785 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782745
15:23:41.758679 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782746
15:23:43.698722 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782747
15:23:45.638704 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782748
15:23:47.579061 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782749
15:23:49.275833 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782750
15:23:50.278633 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782751
15:23:51.288993 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782752
15:23:52.298533 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782753
Backup node shows the same...