Most probably a case of asymmetric routing. Make sure packets travel the same way in both directions.