VPN IPSec through LAN nic has no reply-to

[GTA_doum]

Hello,
I have an IPSec tunnel mounted between two OPNsense routers.  One router is not the main one for internet access for the LAN.  This setup was build before with IPFire and it was working properly.
So the issue is that if I ping from one side to the side where the OPNsense router is only routing VPN, the computer on that side receives the packet but replies to the main router instead of the incoming packet one, which means the packet has no reply-to information!  I tried the option to deactivate Reply-to in OPNsense, but as I would suspect, did not fix the issue.
Any suggestions to have this setup work with a reply-to information in the packet?
Thanks.

[GTA_doum]

Things I've tried and discovered.

To compare, I installed pfSense with the same configuration and the packets sent to the destination computer were containing the right Destination IP, the pfSense one (is this the Reply-To or it is NAT doing the masquerade?).

So on OPNsense, I tried to add manual NAT outbound rules, which modifies the destination IP in the packet, the receiving computer replies to OPNsense, which then looses the packet, it does not seem to know where to return it!  Maybe the manual NAT outbound rule is just confusing it...

Any suggestions as to what to try next?