The Scot-born former cabinet minister yesterday urged him to instead focus efforts on curbing the number of low-skilled workers heading to Britain to stave off the threat from Ukip.
Dr Fox warned the prime minister he would be guilty of "dangerous complacency" if he failed to address the damage Nigel Farage's party could do at the ballot box.
The Eurosceptic called for the party to drop the "statistical nonsense" and develop a "clearer narrative" on immigration in a newspaper interview.
Dr Fox said Mr Farage was "personable and reasonable" and that many of his views were shared by "decent and patriotic people".
It comes after migration experts and rival politicians accused the government of losing its grip on immigration policy as official figures revealed earlier this week that the net flow of migrants into the UK surged year-on-year by more than one-third to 212,000 in the 12 months to September 2013.
Liberal Democrat business secretary Vince Cable said the figures were "good news for British workers" and showed that the Conservative drive to get net migration under 100,000 - which has never been accepted by the LibDem side of the Coalition - was "unworkable".
Meanwhile, the Conservative leader of Guildford Council said that he feared the borough would lose its Tory majority for the first time in 12 years.
In a letter to Mr Cameron, Stephen Mansbridge accused him of undermining the party's chances of winning at the 2015 election.
Conservative chairman Grant Shapps said the pledge on net migration was a matter for towards the end of the current parliament.