NAT/Incoming traffic not working when going to secondary gateway/IP

[GureenRyuu]

Hello all.

We've just installed 2 additional internet connections. We've created a total of 3 VLans, 1 for each WAN, and we've setup the gateways for them.
Outgoing traffic is working great.
Incoming traffic that needs to be natted however, only works on the main gateway IP/interface.
Example:
50.50.50.1 (main gateway)
50.50.51.1 (second gateway)
50.50.52.1 (third gateway)

When I call ip 50.50.50.1 port 443, traffic reaches our web server (Located on LAN) and it responds back normally.
However, when we call 50.50.51.1 or 50.50.52.1 port 443, the web server is unable to reply, and the requests timeout.

If we switch the main gateway, for example make 50.50.51.1 the main gateway, then traffic to it's interface works, and traffic to the other 2 stops.

Any idea why this is happening? Am I missing a configuration?