Mr Hollande waded into the dispute when he offered Leonarda Dibrani, a teenager of Roma origin ordered off a school bus and deported to Kosovo, the chance to return to France, but only if she did so alone.
The offer provoked condemnation, including from Ms Dibrani, who said she did not want to return without her family.
Francois Bayrou, a centrist party leader who ran against Mr Hollande in the 2012 presidential election, said: "80% of French people think the state has totally lost its compass, deciding one thing and then deciding the exact opposite one minute later.
"Mr Hollande's authority is significantly weakened here."
Ms Dibrani's expulsion after her family failed to obtain political asylum has tested Mr Hollande's ability to handle the issue of illegal migration, a source of increasing public frustration in France.
Students protested to demand the 15-year-old schoolgirl be allowed back, even as opinion polls showed most French people wanted the family out.
Opponents from the centre-right UMP party accused Mr Hollande of being so obsessed with satisfying his socialist base that he had betrayed the will of the public.
Even socialists appeared dissatisfied with the president's attempt at a compromise.
Minutes after Mr Hollande's TV appearance on Saturday, in which he said police had followed rules but lacked tact, Socialist Party leader Harlem Desir said Ms Dibrani's family should be let back into France.
A poll in the JDD weekly newspaper showed Mr Hollande's approval rating had sunk to 23%, the lowest level in his presidency and beating the record low popularity ratings of his predecessor Nicolas Sarkozy.