Gateway only works one way (hear me out)

[Underpay6703]

Disclaimer & Goal
I'm a novice hobbyist so do tell me if I make false claims.
I hope to figure out why my gateway interface between my two routers won't work both ways.
Some of the pictures may be a bit wide and require sideways scrolling to see fully.

Setup
2 Routers, each have their own IP from a different ISP, running a CARP setup that is between the poor man's CARP and regular CARP (middle class CARP?). It's not seamless failover, but as close as I could get (switches could not be used to split the ISP connections to each router).


To make sure both ISP connections can be used by the Master, a gateway interface is made between the routers (separate from pfsync interface). Followed by a gateway group and firewall policy routing to use the group.

Since they are "local" gateways, I use DNS addresses as Gateway Group monitor IPs to figure out whether the ISP on the backup router is reachable through the master (picture shown in "Gateway Configuration"). 

Gateway configuration: success on one way config

In order for the master to use its own (gateway interface) source_ip when it communicates with the other router, I need to declare the interface as a gateway in the interface configuration.



This gives me the expected result, a gateway to the internet of backup router.


But because I did not configure the gateway to my master's router on my backup router's interface, his connection is down



The Issue: Gateways down when configured on both ends
While one way works, if I set the gateway rules on the backup when they already set on master, neither connection will now work.
They are both considered "Down" on the gateway monitoring.
If I remove the Gateway Rules option on either router's interface, that side's gateway will work.   

My questions to the forum
1. Is this a bug, or simply how it works?
2. Should I simply create a new physical connection, this time configuring the gateway on router1's side and call it a day?