Danger for prime minister is that the policy is becoming synonymous with a failure to grip illegal migration