CARP Failover with multiple interfaces

[beloc]

Hello guys,

I know that CARP has the ability to do CARP groups (not VHID groups) whereas if one of the members of the group fails, then they all fail regardless of status.  I dont see the ability to do this in OPNsense and this bit me in the tail today. I have to interfaces (LAN and WAN) and I have CARPS on both. The LAN interface is plugged into a switch and the WAN interface is of course plugged into another switch provided by the ISP. My LAN switch had an issue today and the LAN interface failed over to the other OPNsense box on the other switch, however, since the WAN interface switch was fine, the WAN CARPS did not failover causing me to have an issue where traffic was coming into the router on the failed LAN switch but the gateway for the LAN was on the second router which had taken over the CARP for the LAN.. Needless to say, this caused a routing issue.

I assume this needs to be configured:

net.inet.carp.preempt: 1

Am i missing something in the configuration or was this removed?

Beloc

[mimugmail]

Preempt only on the backup peer (requires reboot).
Normally this should be enough ..

[andrewhotlab]

Sorry, but I really need help to understand OPNsense's behavior in this scenario. I'm experiencing the same issue described by @beloc, but only when LAN interface goes down (preemption just works with WAN link down).

With the hope to be able to migrate all our production pfSense clusters, I started testing OPNsense 18.7.4, but in the same simple scenario (identical hardware on both nodes, WAN, LAN and SYNC dedicated interfaces, CARP on both WAN and LAN), OPNsense behaves differently from pfSense, which always preempt when both LAN or WAN interface goes down.

I even tried the same setup with vanilla FreeBSD 11.1, and it behaves exactly like pfSense... thus, even if I was hoping it was my fault, I'm really starting thinking there is something wrong in this release of OPNsense.

I tested with the default settings: net.inet.carp.preempt always set to "1" on both nodes, and only OPNsense fails to preempt WAN's CARP address when LAN links goes down. Obviously, if I disable it on primary node it works, but in that way CARP's addresses never fails back to primary node when it comes back online.

I didn't find anything which could explain this behavior in forum and documentation.

Thanks for any help you might give me!

[mimugmail]

I'll so some intensive tests on Thursday, Just follow this issue

https://github.com/opnsense/core/issues/2780

[andrewhotlab]

Thank you very much Michael. In the mean time I added a 4th interface (DMZ) to both nodes and configured a new CARP vhid on them. The preemption works as expected when both WAN or DMZ links go down, only LAN link failure seems to trigger the issue.

Here after the logs from both nodes when the WAN link goes down on master:

--- MASTER NODE
Oct  4 09:57:15 OPN01 kernel: carp: 2@em0: MASTER -> INIT (hardware interface down)
Oct  4 09:57:15 OPN01 kernel: carp: demoted by 240 to 240 (interface down)
Oct  4 09:57:15 OPN01 kernel: em0: link state changed to DOWN
Oct  4 09:57:15 OPN01 kernel: carp: 3@em2: MASTER -> BACKUP (more frequent advertisement received)
Oct  4 09:57:15 OPN01 kernel: ifa_maintain_loopback_route: deletion failed for interface em2: 3
Oct  4 09:57:15 OPN01 kernel: carp: 1@em1: MASTER -> BACKUP (more frequent advertisement received)
Oct  4 09:57:15 OPN01 kernel: ifa_maintain_loopback_route: deletion failed for interface em1: 3
Oct  4 09:57:16 OPN01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (172.21.3.66 :: )
Oct  4 09:57:16 OPN01 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.100.254 - DMZ-CARP (3@em2)" has resumed the state "BACKUP" for vhid 3
Oct  4 09:57:16 OPN01 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.56.254 - LAN-CARP (1@em1)" has resumed the state "BACKUP" for vhid 1

--- BACKUP NODE
Oct  4 09:57:15 OPN02 kernel: carp: 3@em2: BACKUP -> MASTER (preempting a slower master)
Oct  4 09:57:15 OPN02 kernel: carp: 1@em1: BACKUP -> MASTER (preempting a slower master)
Oct  4 09:57:15 OPN02 kernel: arp: 192.168.56.254 moved from 00:00:5e:00:01:01 to 08:00:27:c3:73:2b on em1
Oct  4 09:57:16 OPN02 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.100.254 - DMZ-CARP (3@em2)" has resumed the state "MASTER" for vhid 3
Oct  4 09:57:16 OPN02 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.56.254 - LAN-CARP (1@em1)" has resumed the state "MASTER" for vhid 1
Oct  4 09:57:19 OPN02 kernel: carp: 2@em0: BACKUP -> MASTER (master timed out)
Oct  4 09:57:19 OPN02 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "172.21.3.65 - WAN-CARP-01 (2@em0)" has resumed the state "MASTER" for vhid 2


And here are the logs when I test the link failure on LAN interface:

--- MASTER NODE
Oct  4 09:53:24 OPN01 kernel: carp: 1@em1: MASTER -> INIT (hardware interface down)
Oct  4 09:53:24 OPN01 kernel: carp: demoted by 240 to 240 (interface down)
Oct  4 09:53:24 OPN01 kernel: em1: link state changed to DOWN
Oct  4 09:53:24 OPN01 kernel: carp: 3@em2: MASTER -> BACKUP (more frequent advertisement received)
Oct  4 09:53:24 OPN01 kernel: ifa_maintain_loopback_route: deletion failed for interface em2: 3
Oct  4 09:53:24 OPN01 kernel: carp: 2@em0: MASTER -> BACKUP (more frequent advertisement received)
Oct  4 09:53:24 OPN01 kernel: ifa_maintain_loopback_route: deletion failed for interface em0: 3
Oct  4 09:53:24 OPN01 opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for lan
Oct  4 09:53:24 OPN01 kernel: ifa_maintain_loopback_route: deletion failed for interface em1: 3
Oct  4 09:53:24 OPN01 kernel: ifa_maintain_loopback_route: deletion failed for interface em1: 3
Oct  4 09:53:24 OPN01 kernel: carp: demoted by -240 to 0 (vhid removed)
Oct  4 09:53:24 OPN01 kernel: em1: promiscuous mode disabled
Oct  4 09:53:25 OPN01 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.100.254 - DMZ-CARP (3@em2)" has resumed the state "BACKUP" for vhid 3
Oct  4 09:53:25 OPN01 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "172.21.3.65 - WAN-CARP-01 (2@em0)" has resumed the state "BACKUP" for vhid 2
Oct  4 09:53:25 OPN01 kernel: carp: 3@em2: BACKUP -> MASTER (preempting a slower master)
Oct  4 09:53:25 OPN01 kernel: carp: 2@em0: BACKUP -> MASTER (preempting a slower master)
Oct  4 09:53:25 OPN01 kernel: arp: 172.21.3.65 moved from 00:00:5e:00:01:02 to 08:00:27:3b:3b:ad on em0
Oct  4 09:53:26 OPN01 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.100.254 - DMZ-CARP (3@em2)" has resumed the state "MASTER" for vhid 3
Oct  4 09:53:26 OPN01 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "172.21.3.65 - WAN-CARP-01 (2@em0)" has resumed the state "MASTER" for vhid 2


--- BACKUP NODE
Oct  4 09:53:24 OPN02 kernel: carp: 3@em2: BACKUP -> MASTER (preempting a slower master)
Oct  4 09:53:24 OPN02 kernel: carp: 2@em0: BACKUP -> MASTER (preempting a slower master)
Oct  4 09:53:24 OPN02 kernel: arp: 172.21.3.65 moved from 00:00:5e:00:01:02 to 08:00:27:db:85:ef on em0
Oct  4 09:53:24 OPN02 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.100.254 - DMZ-CARP (3@em2)" has resumed the state "MASTER" for vhid 3
Oct  4 09:53:25 OPN02 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "172.21.3.65 - WAN-CARP-01 (2@em0)" has resumed the state "MASTER" for vhid 2
Oct  4 09:53:25 OPN02 kernel: carp: 3@em2: MASTER -> BACKUP (more frequent advertisement received)
Oct  4 09:53:25 OPN02 kernel: ifa_maintain_loopback_route: deletion failed for interface em2: 3
Oct  4 09:53:25 OPN02 kernel: carp: 2@em0: MASTER -> BACKUP (more frequent advertisement received)
Oct  4 09:53:25 OPN02 kernel: ifa_maintain_loopback_route: deletion failed for interface em0: 3
Oct  4 09:53:26 OPN02 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.100.254 - DMZ-CARP (3@em2)" has resumed the state "BACKUP" for vhid 3
Oct  4 09:53:26 OPN02 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "172.21.3.65 - WAN-CARP-01 (2@em0)" has resumed the state "BACKUP" for vhid 2
Oct  4 09:53:27 OPN02 kernel: carp: 1@em1: BACKUP -> MASTER (master timed out)
Oct  4 09:53:27 OPN02 opnsense: /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "192.168.56.254 - LAN-CARP (1@em1)" has resumed the state "MASTER" for vhid 1


Obviously I switched interface assignment between WAN and LAN, but no difference. Thus it does not seem to be hardware related.

Analyzing the logs, I noticed that this line can put us on the right path to understand:

OPN01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (172.21.3.66 :: )

It's shown also when DMS interface goes down:

OPN01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for DMZ(opt2) but ignoring since interface is configured with static IP (192.168.100.66 :: )

But when the LAN goes down (it has too its IPv4 address statically configured), here it is:

OPN01 opnsense: /usr/local/etc/rc.linkup: DEVD Ethernet detached event for lan

And this seems to cause vhid to be deleted from LAN interface, which I guess causes the inability to preempt other vhid's:

OPN01 kernel: ifa_maintain_loopback_route: deletion failed for interface em1: 3
OPN01 kernel: ifa_maintain_loopback_route: deletion failed for interface em1: 3
OPN01 kernel: carp: demoted by -240 to 0 (vhid removed)
OPN01 kernel: em1: promiscuous mode disabled


Thanks for your help.

[mimugmail]

Can you add screenshots of HA config and VIP config of both nodes. I'm tomorrow in the office for testing ..

[andrewhotlab]

Sure, you can find screenshots in the attached zip file. I also attached full configuration files, thus you can easily reproduce the environment using virtual machines.

Thanks.

[mimugmail]

I always test on real hardware .. it's way closer to reality Thanks!

[mimugmail]

What happens when you remove carp in LAN, change IP Network in LAN, add a new one with LAN2 and the original IP Network and add carp there? Does it still happen?

[andrewhotlab]

Wow! I removed CARP vhid on LAN, changed the IP on the LAN interface (a couple of time, to set back the original), then re-created CARP vhid and... voila'! :)

Now this happens when I bring LAN interface down on master, and preemption works like any other vhid:

OPN01 opnsense: /usr/local/etc/rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (192.168.56.66 ::)

Thus OPNsense simply didn't "perceived as static" the LAN IP address the first time I set it up? :P

[mimugmail]

You mean I should remove the VIP, set multiple times the IP addresses on LAN, create the VIP again and it works?

[andrewhotlab]

Yes, it's even simpler: remove VIP, change IP on LAN and re-create VIP just works.
Maybe there is a flag "this interface has a static ip" somewhere in OPNsense's frontend configuration which was not set during first configuration of LAN interface?