OPNSense HA failover switching to backup

[siginigin]

Hi guys,

Let's assume this simple scenario: we have 2 firewalls in HA, each has with 4 physical cables, one is direct connection between them for pfsync. CARP seems working, and pfsync is operating.

Now, what happen if I pull out just one cable from master interface. From what I've saw, backup firewall becomes master, but only for this one interface, other interfaces are still backup. So now I have split routing. Client packets behind new master interface are traveling through backup firewall, but returning packets are going through master firewall and there are dropped because of failed interface.

Cisco ASA has failover interface-policy <percent>, which simply says, that if x percent of interfaces fails, firewalls will fail over all interfaces. Is there something similar on OPNSense?

Thank you for explanation.

[mitsos]

That's not how it's supposed to work. There is an issue with your switch upstream of opnsense. In case of a cable being unplugged everything "fails over": the backup (correctly) assumes the master role, notifies your upstream switch (correctly) that your "WAN" IP is now handled by the slave, clients behind your cluster (correctly) see that their gateway's IP (LAN) switched and start (correctly) using that. Return packets arriving at your upstream switch (on opnsense's wan side) go through your switch's routing table and are (wrongly) spit out at the old interface associated with the master's MAC address.

Cisco also has (many) factory backdoors, but we don't want that around here  .

Is the upstream switch a "dumb" switch or a managed switch?

[mimugmail]

Do you have disable preemption on one of the systems ticked?

[siginigin]

Hi,

thanks for reply. Upstream switch is managed cisco and "Disable preempt" is NOT ticked on master, but IS ticked on backup node. I did that because according to help line underneath - when firewall is booted, it will switch to master and I didn't wanted this feature on backup node. Or I did I get it wrong? Should I have it unticked on both nodes?

There is also worth mention that there is one cable missing on backup node - waiting for our networkers to configure it on their switch. So maybe the situation will change if all interfaces are up.

So if I understand correctly - in this case of one cable missing on  backup node, if I unplug cable from master, the master node will not fail over all interfaces, but only the one that failed. Otherwise, with all interfaces up on both nodes, if I unplug cable on master, it will failover all interfaces. Is that correct?

And what part of firewall ensures that all interfaces will failover, is it kernel? AFAIK the carp is running independent on interfaces, so one carp per interface. And on the switch side, both interfaces should see each other, ie. same vlan with no restrictions.

Thank you.

[mimugmail]

Please don't tick it on both systems, reboot both, be happy

[mitsos]

thanks for reply. Upstream switch is managed cisco and "Disable preempt" is NOT ticked on master, but IS ticked on backup node. I did that because according to help line underneath - when firewall is booted, it will switch to master and I didn't wanted this feature on backup node. Or I did I get it wrong? Should I have it unticked on both nodes?

Ah, not even a "dumb" switch, just plain ol' dumb (without the quotes) 

"well, there's your problem" 

unticked preempt as mentioned by mimugmail on both

There is also worth mention that there is one cable missing on backup node - waiting for our networkers to configure it on their switch. So maybe the situation will change if all interfaces are up.

Get that cable plugged in otherwise it will trigger a failover.

So if I understand correctly - in this case of one cable missing on  backup node, if I unplug cable from master, the master node will not fail over all interfaces, but only the one that failed. Otherwise, with all interfaces up on both nodes, if I unplug cable on master, it will failover all interfaces. Is that correct?

no, see above, everything fails over at the same time. In your particular case the switch see the IP>MAC association on both ports, and figures that the "first one must be right, so I'll continue using it".

And what part of firewall ensures that all interfaces will failover, is it kernel? AFAIK the carp is running independent on interfaces, so one carp per interface. And on the switch side, both interfaces should see each other, ie. same vlan with no restrictions.

Does it matter what ensures that all will failover? they will, trust me, I've run this configuration for years. Yes, both members should see each other with no "filtering" (unicast,anycast,anything) between the two on the switch side.

[siginigin]

Hi guys, thank you very much for help & explanation.  I'll do that.