CARP and Multiple Internal Interfaces

[spetrillo]

Hello all,

I am about to embark on clustering two OPNSense nodes and I have a couple questions:

1) I am going to have 3 internal interfaces and a Wireguard interface. Do I need to add a CARP firewall rule to each interface?
2) For the heartbeat network should the firewall rule be wide open or limited to the heartbeat network.

Thanks,
Steve

[crlt]

Hello all,

I am about to embark on clustering two OPNSense nodes and I have a couple questions:

1) I am going to have 3 internal interfaces and a Wireguard interface. Do I need to add a CARP firewall rule to each interface?
2) For the heartbeat network should the firewall rule be wide open or limited to the heartbeat network.

Thanks,
Steve

1) I believe the CARP firewall rule is automatically added and doesn't require manual rule creation. If you did want to make one though you could make a rule like the one below for every interface? Not sure about Wireguard but mine has been working without a custom rule.

 Protocol    Source            Port    Destination    Port    Gateway    Schedule       Description    
IPv4 CARP    carp_vlan10     *            carp_vlan10     *              *                     *

2) Most tutorials I've seen online for CARP create a wide open rule but I don't like that approach because what if somebody plugs in the wrong cable after poweroff/disconnect and they put your WAN link into the pfsync port? What I do is make a rule that allows  SOURCE PFSYNC network to DESTINATION PFSYNC network on both sides and it has worked for me. Maybe somebody else has a suggestion as to a better rule?

Protocol    Source            Port    Destination     Port    Gateway    Schedule
IPv4*    PFSYNC net    *            PFSYNC net     *            *                     *

Disclaimer: I am not an expert and just writing my experience. Perhaps someone more knowledgeable can chime in?

[mimugmail]

Auto rule will handle carp. But for pfsync and config sync you need a manual rule. You dont need a dedicated interface for this if the number of available interface is limited

[spetrillo]

OK making progress...but...

When I go to System/High Availability/Status, on the backup node, I have a message that says the backup firewall is not accessible or not configured. Is this ok? The master node seems to be happy.

On additional thing to note...these nodes are VMware virtual machines.

[Patrick M. Hausen]

When I go to System/High Availability/Status, on the backup node, I have a message that says the backup firewall is not accessible or not configured. Is this ok? The master node seems to be happy.

This is expected. The backup system does not have another backup system. If the active node shows the backup node version, that part is fine.

[spetrillo]

All good then...yes the master node is showing the backup at the top of the status page. I also checked the logs and pfSync seems to be working fine....woo hoo!

One last question...for those who run Wireguard...do you set CARP to the WAN? That is what I did, so if the WAN fails over then Wireguard will go with it.

[mimugmail]

Yes, you need to track it so that when MASTER state switches the wireguard instance on the old master does not disrupt anything.

Also to add to Patricks comment, synchronisation of config is a one-way street. Only from Node1 to Node2.

[spetrillo]

I get it...I just wish there was a more informative way of saying it, or have a dedicated CARP dashboard for the backup node/s that tell you where the master is.

[Patrick M. Hausen]

You mean like the CARP dashboard widget?

[spetrillo]

You mean like the CARP dashboard widget?

I use the CARP dashboard but I would like to see a different one for the backup, indicating where the master is. Right now the dashboard on the secondary node is the same as the primary node. First screenshot is the primary and second screenshot is the secondary.

[Patrick M. Hausen]

Than you have a split-brain or otherwise broken situation. The dashboard on the backup node should read BACKUP 10.0.3.1 etc.

[spetrillo]

Hmmm...I thought that could be the case.

Ok now how to figure out where the split brain is going on. Under System/High Availability/Settings should Disable Preempt be checked on the backup node? I also noticed the advbase on both nodes was the same, so I adjusted it to 100 on the backup node.

[Patrick M. Hausen]

I also noticed the advbase on both nodes was the same, so I adjusted it to 100 on the backup node.

That will most likely fix it IIRC.

[spetrillo]

Seems to have done the trick!

Now one last question. On one guide I read the following:

VMware ESXi: Activate Allow forged transmits and Allow MAC changes. If necessary, Promiscous Mode must also be activated. Additionally, the os-vmware plugin can be installed.

Is this still true? Reason I ask is I am using a VMware cloud provider and they have told me they cannot enable these for me. These would have to be enable on each hypervisor, for all clients.