OSPF vs CARP vs Gateway Groups

[clarknova]

Our goal is to set up a VPN link between two sites with a pair of OPNsense firewalls at each site acting as VPN endpoints. We are leaning toward Wireguard as our VPN protocol, but that's not the focus of this post.

Diagram:

           /------OPNsense A1--------OPNsense B1------\
LANs A-----                      X                     ------LANs B
           \------OPNsense A2--------OPNsense B2------/

Goals:

• LANs A and B should have bilateral (non-NAT) IPv4 communication.
• Firewalls should share LAN routes between A and B side.
• Connectivity should be able to tolerate the failure of one firewall on each side with minimal convergence / failover period.

Questions:

• Our initial thought was to use CARP for failover and OSPF for route propagation. While I think CARP makes sense on the LAN interfaces, I am wondering if it makes sense to skip CARP on the A-B link. Can OSPF provide that redundancy / failover given that it will be active on the VPN interface anyway?
• Should I be looking at gateway groups for failover rather than CARP or OSPF for the A-B link? I'm not really sure what the overlap is between these protocols and I haven't found any documentation that really makes it clear to me what the interplay would be in a setup like this.

[fabian]

With carp both firewalls share a single IP address. This simplifies the problem, that both firewalls can be the default gateway. This provides redundancy in a single LAN. On the other hand, OSPF is a routing protocol. If your network changes, it will regenerate the routing tables. This provides redundancy on L3 if it is possible.

You can also load balance and fail over by having two routes with the same metric (load balance) or a different one (fail over).

[clarknova]

With carp both firewalls share a single IP address.

I think CARP is the obvious choice on the LAN sides for that reason. I'm less sure that I need it on the inter-router links where OSPF will be active regardless due to frequent changes on the LAN routes.

My main concern is whether there are gotchas with using CARP on one interface and OSPF on another.

You can also load balance and fail over by having two routes with the same metric (load balance) or a different one (fail over).

How well does this work when OSPF is balancing and CARP is in failover mode? For example, suppose A1 and B1 are both CARP master on their LAN and OSPF is balancing traffic between A and B sides. I expect to see all LAN A traffic coming from A1 and being forwarded to both B1 and B2 to be forwarded to LAN B. Return traffic from LAN B is picked up by B1 and forwarded to A1 and A2. Does pfsync manage this asymmetry in a way that is conducive to a good experience for network users?

And what happens if the OSPF peer goes down on the CARP master? Will OPNsense recognise that those upstream OSPF gateways are down and demote itself on the LAN CARP?

I am going to test some or all of these modes, but I thought I'd see if there was any wisdom floating in the community that could save me going down any blind alleys.

[mimugmail]

I did this on customer side some months ago with BGP, it also worked with a local HAProxy bound to dmz which was announced outside