OPNsense Firewalls Crashes in HA mode

[u63725]

Hi, I have two OPNsense Firewalls in HA Cluster. Both Firewalls have two ssd's in Raid 0 and runs on version 20.7.3. The Problem is, that firewall 1 hangs up after a half hour. After that firewall 2 becomes master and hangs also up after a half hour. There is no error in Logfiles or on the Screen. I removed one older SSD in firewall one. I could not find the reason for this problem.

[mimugmail]

Raid0 in hardware Controller or Raid1 geom mirror? Maybe better Install on a clean setup with only one ssd

[u63725]

Raid1 geom mirror

[mimugmail]

Troubleshooting would look like this:

1) In cluster put a display on both units and watch for stack traces and collect logs
2) cut the cluster and let both units just run to see if they are alive for more than one day
3) User one FRESH ssd on each node, reinstall and restore config

[u63725]

After Firewall one hangup Firewall two hangs also up after some minutes. On Firewall one is no error message on the screen, on Firewall two is an arp error:

https://imgur.com/a/SSmkGvk

[mimugmail]

Next test, both units without cable attached, wait if it hangs

[u63725]

Should all data cables be disconnected from the firewalls, or should only the sync cable be left on? It should be mentioned that I only allowed CARP in the firewall rules for the sync interface.

[mimugmail]

And what about config sync? If it's a direct connection you can safely allow everything

[u63725]

Yes it is a direct connection for sync. The communication between the firewalls is established. On the Ubiquity Core Switch is Rapit Spanning tree enabled. When I disconnect WAN and the Vlan Uplink on both firewalls, direct connection for Sync is still connected there are no hangups on both Firewalls. On the WAN interface is DHCP enabled and on the LTE Modem side both Firewalls have a static ip address assingment.

[mimugmail]

DHCP and carp ha doesnt really work well

[u63725]

DHCP on WAN or on LAN?

[mimugmail]

DHCP in WAN. But for me it seems you have a loop somewhere and not related to hardware

[u63725]

Is it recomendet to enable rstp on WAN and LAN Switch? How can I find the loop, are there some specific messages in the log file

[mimugmail]

Only on the Switch .. but Unify are veeeery bad at logging

[u63725]

https://imgur.com/a/GIr5LTg This are the Ubiquity Core Switch settings. I set the Priority to 40960. Does CARP have to be allowed over the firewall rules for the WAN and LAN interfaces?