OpenVPN access to same LAN from diferrent Firewalls

[Nasospan]

The setup as you see in the attachment is the following:

- 2 Data Centers are connected on the same layer 2 domain.
- In each datacenter we have one ISP connected to the WAN port on the opnsense firewalls. LAN ports of the firewalls are connect to the same layer 2 domain switches.
- All servers are directly attached to these. Openvpn has been configured on both sides and provides access to users to the 172.16.16.0/24 network. Each Openvpn server provides his own network to the clients. 10.10.10.0/24 and 10.10.20.0/24.
- Servers have configured randomly their default gateways to either one or the other firewall (172.16.16.1 or 172.16.16.254). After all everything is a layer 2 domain and we are free to decide how to route traffic with a simple GW change.

What I want to achieve is whether a user connects to DC1 or DC2 openvpn to be able to access all servers regardless of their gw configuration.

The issue arises when a user connects to DC1 Openvpn (lets say he got the 10.10.10.10 ip) and wants to access the server with IP 172.16.16.100 and 172.16.16.254 configured gateway. Server gets packages but the reply to an unknown network for it (10.10.10.0/24) will go through its GW. The GW (Firewall with .254 GW) does not now ofc where to route the traffic (it knows only his openpvn network 10.10.20.0).

-> What I tried to do firstly is to add static routes to the firewalls to reach the other firewalls vpn network from the correct way (create a gateway with the other firewall's lan address and add a static route). It didn't work
I then tried manual routes from shel directly to the ethernet device. It didn't worked

-> I also tried to add the other network on the OpenVPN configuration "IPV4 Remote Network" field. It didn't work.

-> I tried to assign an Ip from the openvpn range network to the other firewall's LAN as an alias to learn this network. Again it didn't work.

-> I tried to switch Openpvn configuration from tun to tap. Again it didnt work.

-> I tried to enable/disable the hardware checksum offload. It didn't help.

What 's really interesting is that from a server (172.16.16.0/24) I can initiate a proper connection with any openvpn client on any firewall!!! I cannot do the opposite.

I am confident that in all of my checks my configuration is correct. I can share ofc configurations.

I have this setup also in a virtual environment where I enabled in/out any/any rules on all interfaces. It's not a firewall rule issue.

Can anyone guess what's going on?

The only way I thought to bypass this issue is to add custom routes by hand on the servers. This way I will be sure that traffic to openvpn networks will be routed directly to the GW without the need to pass from another firewall. I tried it and it works buts its not easy to maintain routes on many machines. Also there are services that I want to access that cant handle routing (like IPMI and BMC).