Ok, so the ARP issue was due to OPNsense bridge I had configured on 3 ports getting all confused where to forward the ARP reply to it seems. As soon as I removed the bridge and used a single port instead, it all went away. Best to stay away from bridging and use a separate switch instead I'd say. Perhaps this may help someone in the future.