CARP - OPNSense slow!

[remd]

I am configuring 2 other OPNSense Appliances, running 18.1.6, and I've followed the documentation on how to setup CARP, and have set it up as I have done on 2 other pair of systems.

This time however as soon as I enabled CARP the 2 systems became very slow. They have a dedicated interface and cable connected between them, they can ping eachother fine, but as soon as I change something on the main system its spinning for about 10 minutes before the change is reflected on the other appliance and before I can do anything else on the system, so its very painful.

I tried to look into the logs, but...it takes about 10min of waiting after I do something, and then the logs seem to have changed since v17x..., so not sure where to find all the infos. That said the CARP communication are shown in Green, its just very slow.
The system reacts normally again if I disable CARP.

Does anyone had a similar experience or any idea what can cause this ?

 

[doug.dimick]

That sounds like it could be an issue with XMLRPC. Are you using the CARP VIP IP address for anything in the System/High Availability setup?

[remd]

I have just setup the CARP VIP's in the NAT outbound rules, but nothing else in the HA setup.
I'm starting to suspect that the upgrade to 18.1.6 has corrupted some of the CARP/HA configuration, as the HA conf was gone on one of the appliances after the upgrade and the outbound rules seem to have been set back to int instead of the vip. And maybe other things I haven't noticed yet as the system is really slow.
The command line isn't slow however, so looking through the logs, but I'll probably try to recover from a backup and hopefully it will help.

[Headologic]

I think, it's the same i posted some time ago. You are not alone...
https://forum.opnsense.org/index.php?topic=6496.msg27888#msg27888

[Headologic]

pump...I don't usually do this, but I think the problem should be solved.
The problem already existed on pfSense.
What information do you need to find the problem?

[remd]

Another observation, if you turn the second firewall off, make the changes on the main firewall, then start the 2nd firewall it will update all the changes through CARP and the systems remain fast.
However if you make any change while both are up it will take about 5 Min to replicate the change to the other firewall.

The problem remains with 18.1.8, and I have it on all 6 firewalls now, so its a real pain!

[remd]

I should add that the entire web interface becomes slow and is unusable when making any change, the command line remains ok.
And everything is ok again after a reboot, but this means that if you want to make any changes the second firewall has to be turned off, then switched on again when the changes are made, they will then be replicated on the other firewall and the systems remains responsive.
It is however a pain to do so when the firewall is in a datacenter and you have to go there to switch them back on (I tried to setup the AMT/IPMI access but couldn't do so in a secure way (ssl) so I disabled it).
So when I have just a few changes I'll reboot the second one and make the changes while its rebooting, I usually have to reboot a few times to give enough time to update some rules, with the risk of making both systems unresponsive if the changes take too long and have to reboot both to get a responsive system again.... you see the picture, its a real pain and it would be great to know why this is happening and to fix it!

[mimugmail]

Hi,

this problem is definitive just on your side and not a general problem.
Can you check via tcpdump on the syncing interface what happens when you configure things?

[hellfirehd]

We are experiencing similar issues.  I'm running OPNsense 18.1.13-amd64 on two identically configured HP DL360 G5 (OFW-A and OFW-B) configured for high availability.  Interfaces are named the same with the same assignments as follows:

WAN (em0)
LAN (em1)
CARP (bce2)

OFW-A CARP is plugged directly into OFW-B CARP with no switch involved.

In order to get pfSync to work we had to add an Allow rule to the CARP interface on OFW-B.  As long as that rule is present, the web interface on OFW-A is responsive.  However, the allow rule on OFW-B disappears all the time breaking the HA and the killing OFW-A web interface performance.

[c-mu]

I have had seen the same problem in my setup. It turned out, that there were missing FW Rules between the CARP interfaces. Did you proof that?

[hellfirehd]

It turned out, that there were missing FW Rules between the CARP interfaces. Did you proof that?

Yes.  We add a Pass rule to the CARP interfaces on both firewalls but it disappears on the slave, sometimes within seconds after creating it.

[mimugmail]

Can you check the system.log when this happens?