CARP maintenance mode - not working

[GreenMatter]

Opnsense version is 26.1.3. When I activate in gui carp maintenance mode, almost nothing happens.
Demotion works:

Code Select Expand

sysctl net.inet.carp.demotion
net.inet.carp.demotion: 240
But advskew is not changed on master (regular setting is 1):

Code Select Expand

ifconfig | grep carp
    carp: MASTER vhid 99 advbase 1 advskew 1
    carp: MASTER vhid 16 advbase 1 advskew 1
    carp: MASTER vhid 166 advbase 1 advskew 1
    carp: MASTER vhid 1 advbase 1 advskew 1
    carp: MASTER vhid 16 advbase 1 advskew 1
    carp: MASTER vhid 17 advbase 1 advskew 1
    carp: MASTER vhid 176 advbase 1 advskew 1
    carp: MASTER vhid 11 advbase 1 advskew 1
    carp: MASTER vhid 116 advbase 1 advskew 1
    carp: MASTER vhid 12 advbase 1 advskew 1
    carp: MASTER vhid 126 advbase 1 advskew 1
    carp: MASTER vhid 13 advbase 1 advskew 1
    carp: MASTER vhid 136 advbase 1 advskew 1
    carp: MASTER vhid 14 advbase 1 advskew 1
    carp: MASTER vhid 146 advbase 1 advskew 1
    carp: MASTER vhid 15 advbase 1 advskew 1

Only when I disable carp or shutdown master, backup is elevated to master mode...
I've seen it it's an old issue/bug. Is there any way to fix it?

EDIT:
Results of tcpdump command for one of interfaces of master node when enabling maintenance mode:

Code Select Expand

tcpdump -ni vlan01 -T carp carp

Code Select Expand

15:23:17.637939 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782732
15:23:18.648116 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782733
15:23:19.658610 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782734
15:23:20.420473 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782735
15:23:22.358801 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782736
15:23:24.298404 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782737
15:23:26.237935 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782738
15:23:28.178964 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782739
15:23:30.118701 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782740
15:23:32.058947 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782741
15:23:33.998853 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782742
15:23:35.938963 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782743
15:23:37.878589 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782744
15:23:39.818785 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782745
15:23:41.758679 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782746
15:23:43.698722 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782747
15:23:45.638704 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782748
15:23:47.579061 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=17002279201346782749
15:23:49.275833 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782750
15:23:50.278633 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782751
15:23:51.288993 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782752
15:23:52.298533 IP 172.16.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=1 authlen=7 counter=17002279201346782753
Backup node shows the same...

[GreenMatter]

And today I noticed that backup node is completely unresponsive. In serial console I could see info: 

• pf states limit reached
• and something about swap running out of space

So I guess that somehow RAM got full (while in master node is/was at around 20% of utilisation). Both VMs have assigned 8 GB of RAM.

The only thing I was trying to do, since yesterday reboot, was to activate carp maintenance mode in master node...

[GreenMatter]

Ok now works after enabling preemption and group interface failover by adding tunable: 

• Code Select Expand
  net.inet.carp.preempt=1


To either both nodes or master and sync config. BTW, it is not present in docs: https://docs.opnsense.org/manual/how-tos/carp.html#configure-carp

Now, maintenance failover is instant!

[Monviech (Cedrik)]

The documentation is correct. In a default CARP setup where the environment works correctly, you don't need any tunables.

If you need that you have other issues to solve, check out the troubleshooting guide we wrote for CARP:
https://docs.opnsense.org/manual/how-tos/carp.html#troubleshooting

[GreenMatter]

The documentation is correct. In a default CARP setup where the environment works correctly, you don't need any tunables.

For sure I'm not entitled to say otherwise but...
Demotion/maintenance mode didn't work but after disabling carp or shutting down master node failover worked.
I did check using tcpdump and multicast worked, I had the same results with unicast (setting peer IPs). So I don't know what else could have been wrong to prevent maintenance mode from working (dedicated interfaces connected via also dedicated ovs switch)?
That tunable I found on: https://www.openbsd.org/faq/pf/carp.html ....

And to stop that pfsync storm (caused pfsync states limit reached) which kept happening on backup node after failover master - backup - master, I had to enable on both nodes "Defer pfsync" option.