[Solved] 19.7.3: Fallback to Master HA member not working

[rainerle]

Hi,

after upgrading the Master HA member and a final reboot I clicked "Leave Persistent CARP Maintenance Mode". The Backup HA member would still be Master.

Checking on the WebIF I could see that adskew on the Backup would be 100 and on the Master 0 for all the VHIDs. On the status page the Master showed for "Current CARP demotion level" 0 and on the Backup 0 as well.

Looking on the CLI:
Master:
ixl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 7c:d3:0a:d8:34:a0
        hwaddr 7c:d3:0a:d8:34:a0
        inet 10.20.38.41 netmask 0xffffff00 broadcast 10.20.38.255
        inet 10.20.38.1 netmask 0xffffff00 broadcast 10.20.38.255 vhid 1
        inet6 fe80::7ed3:aff:fed8:34a0%ixl0 prefixlen 64 scopeid 0x1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
        status: active
        carp: BACKUP vhid 1 advbase 1 advskew 254
...

After clicking "Temporarily Disable CARP" and then "Enable CARP" on the Master it just failed back.

After clicking "Temporarily Disable CARP" and then "Enable CARP" on the Backup it stayed on the Master.

Now the Master shows:
WebIF: Current CARP demotion level: -240
CLI: carp: MASTER vhid 1 advbase 1 advskew 0

And the Backup:
WebIF: Current CARP demotion level: 0
CLI: carp: MASTER vhid 1 advbase 1 advskew 0
carp: BACKUP vhid 1 advbase 1 advskew 100

No idea if this is reproducable...

[adrianschneider]

Hi,

I had the same problem and fixed it with the following tuneables:

net.inet.carp.preempt = 1
net.inet.carp.senderr_demotion_factor = 0
net.pfsync.carp_demotion_factor = 0

There is a pfsync thread to this topic.

Hope it works for you!

[rainerle]

So according to Ad Schellevis in
https://github.com/opnsense/core/issues/1100
setting these values to 0 is not advisable.

Is it enough to set preempt to 1 using the checkbox in system_hasync.php?
Is this synchronised to the Backup HA partner or do I have to set it manually on both?
Could we have a sane default HA setup preconfigured?

I like the way Kodi works with its configuration. You select your skill level and Kodi shows the available options accordingly. I am lazy, so I leave it on beginner and everything just works out of the box...

[mimugmail]

Seems you need Sophos If you are lazy
Preempt is default on and a change requires a reboot.
19.7.3 added some changes

[adrianschneider]

He just writes that these values shouldn't be the defaults. But can be changed when needed.

It solved my problem, so I'm fine with it.

[mimugmail]

Hitting the checkbox will disable it, so unchecked is the correct setting (and default)

[rainerle]

I do not think I am lazy, I just want sane defaults for in my humble opinion standard scenarios 

So the checkbox in the WebIF is currently unticked, which gives

Master HA member:

Code: [Select]

root@opnsense01:~ # sysctl -a | grep carp
net.inet.carp.ifdown_demotion_factor: 240
net.inet.carp.senderr_demotion_factor: 0
net.inet.carp.demotion: -240
net.inet.carp.log: 1
net.inet.carp.preempt: 1
net.inet.carp.allow: 1
net.pfsync.carp_demotion_factor: 240
root@opnsense01:~ #

Backup HA member:

Code: [Select]

root@opnsense02:~ # sysctl -a | grep carp
net.inet.carp.ifdown_demotion_factor: 240
net.inet.carp.senderr_demotion_factor: 0
net.inet.carp.demotion: 0
net.inet.carp.log: 1
net.inet.carp.preempt: 1
net.inet.carp.allow: 1
net.pfsync.carp_demotion_factor: 240
root@opnsense02:~ #

Where does the -240 on the Master come from? Is that caused by using LACP?

[mimugmail]

Maybe, can you disable carp in the machine and reboot? Maybe an Error during 19.7.3 migration

[rainerle]

I executed a switch cluster firmware upgrade yesterday and even though we use LACP to the firewalls there was a downtime.

It was actually a design fault:

• We have four interfaces.
• Since it is complex during CLI installation to setup lagg and VLANs the ixl0 on opnsense1 is a "plain" interface attached to switch1. The switch takes care of the VLAN. So this interface gets the LAN assigned and is used to configure opnsense via the WebIF. ixl0 on opnsense2 is attached to switch2. The network does have a gateway and therefore the opnsense is providing a CARP gateway IP.
• ixl1 is connected to switch1 and ixl2 is connected to switch2 - these are the lagg interfaces using LACP and have all the other VLANs configured.
• ixl3 is the pfSync interface.

All fine until you configure a CARP on the VLANs and ixl0 for the gateway IPs. As the port on ixl0 goes down during switch cluster upgrade there is a failover of the firewall.

Then the second switch goes down. failover again.

And as soon the second switch came up again, there was some problem and then a split brain on the firewalls CARP interfaces...

I just redid the whole design and configuration:

• ixl0: admin port in a new VLAN network with no routing and therefore no gateways, etc.
• ixl1,ixl2: lagg with LACP and all the CARPs
• ixl3: pfSync.

As soon as this is done I play around with the Fallback again....

[mimugmail]

Problem with OPN or FreeBSD is that when one member is affected in a lagg, carp Event is triggered. Better way to lacp on one switch but every unit in separate switch.

[rainerle]

Ah, just saw that this is a duplicate of
https://forum.opnsense.org/index.php?topic=13987.0

[rainerle]

So after the reconfiguration of the interfaces I did a reboot one-by-one the switch cluster members.

Switch Cluster (Huawei CloudEngine 6800):

Code: [Select]

<muc-switch10-00>display stack configuration
Oper          : Operation
Conf          : Configuration
*             : Offline configuration
Isolated Port : The port is in stack mode, but does not belong to any Stack-Port

Attribute Configuration:
----------------------------------------------------------------------------
 MemberID      Domain         Priority       Switch Mode     Uplink Port   
Oper(Conf)   Oper(Conf)      Oper(Conf)      Oper(Conf)      Oper(Conf)     
----------------------------------------------------------------------------
1(1)         10(10)          150(150)        Auto(Auto)      4*40GE(4*40GE)
2(2)         10(10)          100(100)        Auto(Auto)      4*40GE(4*40GE)
----------------------------------------------------------------------------

Stack-Port Configuration:
--------------------------------------------------------------------------------
Stack-Port      Member Ports                                                   
--------------------------------------------------------------------------------
Stack-Port1/1   40GE1/0/3           40GE1/0/4                                   
Stack-Port2/1   40GE2/0/3           40GE2/0/4                                   
--------------------------------------------------------------------------------
<muc-switch10-00>reset slot 2
Warning: The slot 2 will reset. Continue? [Y/N]:y
Warning: Board 2 is being reset.

...waiting till standby comes up again...

<muc-switch10-00>display switchover state
   Switchover State  :  Ready
   Switchover Policy :  Board Switchover
   MainBoard         :  1
   SlaveBoard        :  2
<muc-switch10-00>reset slot 1
Warning: Resetting the board in slot 1 may cause system reboot while the switchover state is not ready. Continue?  [Y/N]:y
Warning: Board 1 is being reset.
<muc-switch10-00>Write failed: Broken pipe

... wait till master comes up again ...


OPNsense Master:

Code: [Select]

Sep 16 23:07:28 opnsense01 kernel: ixl2: link state changed to DOWN
Sep 16 23:07:31 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2038)
Sep 16 23:07:31 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2026)
Sep 16 23:07:31 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2040)
Sep 16 23:07:31 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2028)
Sep 16 23:07:31 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan3)
Sep 16 23:07:31 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2020)
Sep 16 23:07:31 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan30)
Sep 16 23:07:31 opnsense01 kernel: carp: demoted by 0 to 0 (send error 50 on lagg0_vlan2027)
Sep 16 23:11:10 opnsense01 kernel: ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 16 23:11:10 opnsense01 kernel: ixl2: link state changed to UP
Sep 16 23:23:36 opnsense01 kernel: ixl0: link state changed to DOWN
Sep 16 23:23:36 opnsense01 kernel: ixl1: link state changed to DOWN
Sep 16 23:23:37 opnsense01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for admin_port(lan) but ignoring since interface is configured with static IP (10.11.10.11 ::)
Sep 16 23:27:13 opnsense01 kernel: ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 16 23:27:13 opnsense01 kernel: ixl1: link state changed to UP
Sep 16 23:27:13 opnsense01 kernel: ixl0: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 16 23:27:13 opnsense01 kernel: ixl0: link state changed to UP
Sep 16 23:27:15 opnsense01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for admin_port(lan) but ignoring since interface is configured with static IP (10.11.10.11 ::)
Sep 16 23:27:15 opnsense01 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ixl0'
Sep 16 23:27:15 opnsense01 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.11.10.11) (interface: admin_port[lan]) (real interface: ixl0).
Sep 16 23:27:15 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'lan'
Sep 16 23:27:15 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Sep 16 23:27:15 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Sep 16 23:27:15 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to wan
Sep 16 23:27:15 opnsense01 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Sep 16 23:27:16 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 2001:4860:4860::8888 via <Removed>
Sep 16 23:27:16 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 2001:4860:4860::8888 via <Removed>
Sep 16 23:27:16 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 8.8.8.8 via <Removed>
Sep 16 23:27:16 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 8.8.8.8 via <Removed>
Sep 16 23:27:21 opnsense01 opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface admin_port.

So there is no failover anymore of the OPNsense cluster, but:
- IPsec tunnels are down
- (maybe because of) Uplink over the CARP IP is not working

After selecting option 11 on the CLI (Reload all services) the master just rebooted, the backup took over and then the master came back up and did a fallback...

So at least the fallback is working, but the master OPNsense should be able to sustain one-by-one reboots of the switch cluster...

Any ideas?

...aaaah, there is a crash report in the WebIF... just sent it...

----
OPNsense Slave

Code: [Select]

Sep 16 23:07:28 opnsense02 kernel: ixl0: link state changed to DOWN
Sep 16 23:07:28 opnsense02 kernel: ixl2: link state changed to DOWN
Sep 16 23:07:29 opnsense02 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for admin_port(lan) but ignoring since interface is configured with static IP (10.11.10.12 ::)
Sep 16 23:07:31 opnsense02 kernel: carp: 1@lagg0_vlan2038: BACKUP -> MASTER (master timed out)
Sep 16 23:07:31 opnsense02 kernel: carp: 3@lagg0_vlan2026: BACKUP -> MASTER (master timed out)
Sep 16 23:07:31 opnsense02 kernel: carp: 5@lagg0_vlan2028: BACKUP -> MASTER (master timed out)
Sep 16 23:07:31 opnsense02 kernel: carp: 2@lagg0_vlan2020: BACKUP -> MASTER (master timed out)
Sep 16 23:07:31 opnsense02 kernel: carp: 7@lagg0_vlan2040: BACKUP -> MASTER (master timed out)
Sep 16 23:07:31 opnsense02 kernel: carp: 8@lagg0_vlan3: BACKUP -> MASTER (master timed out)
Sep 16 23:07:31 opnsense02 kernel: carp: 6@lagg0_vlan30: BACKUP -> MASTER (master timed out)
Sep 16 23:07:31 opnsense02 kernel: carp: 4@lagg0_vlan2027: BACKUP -> MASTER (master timed out)
Sep 16 23:07:32 opnsense02 kernel: carp: 2@lagg0_vlan2020: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:07:32 opnsense02 kernel: carp: 4@lagg0_vlan2027: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2020: 3
Sep 16 23:07:32 opnsense02 kernel: carp: 5@lagg0_vlan2028: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2027: 3
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2028: 3
Sep 16 23:07:32 opnsense02 kernel: carp: 1@lagg0_vlan2038: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2038: 3
Sep 16 23:07:32 opnsense02 kernel: carp: 8@lagg0_vlan3: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:07:32 opnsense02 kernel: carp: 3@lagg0_vlan2026: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:07:32 opnsense02 kernel: carp: 7@lagg0_vlan2040: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2026: 3
Sep 16 23:07:32 opnsense02 kernel: carp: 6@lagg0_vlan30: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2040: 3
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan3: 3
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan30: 3
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan3: 3
Sep 16 23:07:32 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan3: 3
Sep 16 23:11:10 opnsense02 kernel: ixl0: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 16 23:11:10 opnsense02 kernel: ixl0: link state changed to UP
Sep 16 23:11:10 opnsense02 kernel: ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 16 23:11:10 opnsense02 kernel: ixl2: link state changed to UP
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for admin_port(lan) but ignoring since interface is configured with static IP (10.11.10.12 ::)
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ixl0'
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.11.10.12) (interface: admin_port[lan]) (real interface: ixl0).
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'lan'
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to wan
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 2001:4860:4860::8888 via <Removed>
Sep 16 23:11:11 opnsense02 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 2001:4860:4860::8888 via <Removed>
Sep 16 23:11:12 opnsense02 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 8.8.8.8 via <Removed>
Sep 16 23:11:12 opnsense02 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 8.8.8.8 via <Removed>
Sep 16 23:11:17 opnsense02 opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface admin_port.
Sep 16 23:23:36 opnsense02 kernel: ixl1: link state changed to DOWN
Sep 16 23:23:39 opnsense02 kernel: carp: 4@lagg0_vlan2027: BACKUP -> MASTER (master timed out)
Sep 16 23:23:39 opnsense02 kernel: carp: 2@lagg0_vlan2020: BACKUP -> MASTER (master timed out)
Sep 16 23:23:39 opnsense02 kernel: carp: 8@lagg0_vlan3: BACKUP -> MASTER (master timed out)
Sep 16 23:23:39 opnsense02 kernel: carp: 8@lagg0_vlan3: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:23:39 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan3: 3
Sep 16 23:23:39 opnsense02 kernel: carp: 2@lagg0_vlan2020: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:23:39 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan3: 3
Sep 16 23:23:39 opnsense02 kernel: carp: 4@lagg0_vlan2027: MASTER -> BACKUP (more frequent advertisement received)
Sep 16 23:23:39 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2020: 3
Sep 16 23:23:39 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan3: 3
Sep 16 23:23:39 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2027: 3
Sep 16 23:27:13 opnsense02 kernel: ixl1: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 16 23:27:13 opnsense02 kernel: ixl1: link state changed to UP
Sep 17 00:02:45 opnsense02 kernel: carp: 7@lagg0_vlan2040: BACKUP -> MASTER (master timed out)
Sep 17 00:02:45 opnsense02 kernel: carp: 4@lagg0_vlan2027: BACKUP -> MASTER (master timed out)
Sep 17 00:02:45 opnsense02 kernel: carp: 5@lagg0_vlan2028: BACKUP -> MASTER (master timed out)
Sep 17 00:02:45 opnsense02 kernel: carp: 6@lagg0_vlan30: BACKUP -> MASTER (master timed out)
Sep 17 00:02:45 opnsense02 kernel: carp: 2@lagg0_vlan2020: BACKUP -> MASTER (master timed out)
Sep 17 00:02:45 opnsense02 kernel: carp: 3@lagg0_vlan2026: BACKUP -> MASTER (master timed out)
Sep 17 00:02:45 opnsense02 kernel: carp: 1@lagg0_vlan2038: BACKUP -> MASTER (master timed out)
Sep 17 00:02:45 opnsense02 kernel: carp: 8@lagg0_vlan3: BACKUP -> MASTER (master timed out)
Sep 17 00:05:13 opnsense02 kernel: ixl3: link state changed to DOWN
Sep 17 00:05:13 opnsense02 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for pfSync(opt8) but ignoring since interface is configured with static IP (10.50.20.12 ::)
Sep 17 00:05:15 opnsense02 kernel: ixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 17 00:05:15 opnsense02 kernel: ixl3: link state changed to UP
Sep 17 00:05:16 opnsense02 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for pfSync(opt8) but ignoring since interface is configured with static IP (10.50.20.12 ::)
Sep 17 00:05:16 opnsense02 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ixl3'
Sep 17 00:05:16 opnsense02 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.50.20.12) (interface: pfSync[opt8]) (real interface: ixl3).
Sep 17 00:05:16 opnsense02 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for pfSync(opt8) but ignoring since interface is configured with static IP (10.50.20.12 ::)
Sep 17 00:05:16 opnsense02 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ixl3'
Sep 17 00:05:16 opnsense02 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.50.20.12) (interface: pfSync[opt8]) (real interface: ixl3).
Sep 17 00:06:17 opnsense02 kernel: ixl3: link state changed to DOWN
Sep 17 00:06:17 opnsense02 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for pfSync(opt8) but ignoring since interface is configured with static IP (10.50.20.12 ::)
Sep 17 00:06:18 opnsense02 kernel: ixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: None, Autoneg: False, Flow Control: None
Sep 17 00:06:18 opnsense02 kernel: ixl3: link state changed to UP
Sep 17 00:06:18 opnsense02 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for pfSync(opt8) but ignoring since interface is configured with static IP (10.50.20.12 ::)
Sep 17 00:06:19 opnsense02 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ixl3'
Sep 17 00:06:19 opnsense02 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.50.20.12) (interface: pfSync[opt8]) (real interface: ixl3).
Sep 17 00:07:33 opnsense02 kernel: carp: 8@lagg0_vlan3: MASTER -> BACKUP (more frequent advertisement received)
Sep 17 00:07:33 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan3: 3
Sep 17 00:07:33 opnsense02 kernel: carp: 7@lagg0_vlan2040: MASTER -> BACKUP (more frequent advertisement received)
Sep 17 00:07:33 opnsense02 kernel: carp: 3@lagg0_vlan2026: MASTER -> BACKUP (more frequent advertisement received)
Sep 17 00:07:33 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2040: 3
Sep 17 00:07:33 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan3: 3
Sep 17 00:07:33 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2026: 3
Sep 17 00:07:33 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan3: 3
Sep 17 00:07:33 opnsense02 kernel: carp: 6@lagg0_vlan30: MASTER -> BACKUP (more frequent advertisement received)
Sep 17 00:07:33 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan30: 3
Sep 17 00:07:33 opnsense02 kernel: carp: 1@lagg0_vlan2038: MASTER -> BACKUP (more frequent advertisement received)
Sep 17 00:07:33 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2038: 3
Sep 17 00:07:33 opnsense02 kernel: carp: 4@lagg0_vlan2027: MASTER -> BACKUP (more frequent advertisement received)
Sep 17 00:07:33 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2027: 3
Sep 17 00:07:34 opnsense02 kernel: carp: 5@lagg0_vlan2028: MASTER -> BACKUP (more frequent advertisement received)
Sep 17 00:07:34 opnsense02 kernel: carp: 2@lagg0_vlan2020: MASTER -> BACKUP (more frequent advertisement received)
Sep 17 00:07:34 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2028: 3
Sep 17 00:07:34 opnsense02 kernel: ifa_maintain_loopback_route: deletion failed for interface lagg0_vlan2020: 3