Qatada, who has been fighting extradition for more than a decade, was due to be released from prison today after winning an appeal against deportation to Jordan to face trial.
The Special Immigration Appeals Commission (Siac) allowed his appeal and granted him bail, meaning Qatada will be released to return to his home address.
He will be subject to a 16-hour curfew and allowed out between 8am and 4pm, with the condition he wears an electronic tag, does not use the internet, and does not contact certain people.
Once described as Osama bin Laden's right-hand man in Europe, Qatada was convicted of terror charges in Jordan in his absence in 1999.
The country has given the Home Secretary assurances that no evidence gained through torture will be used against him. But in their ruling, Siac judges said they could not be sure this would be the case.
The Government now plans to appeal. Mrs May told the Commons she believed Mr Justice Mitting applied the "wrong legal test". She added: "The Government has been doing everything it can to get rid of Abu Qatada and we will continue to do so."
Qatada featured in hate sermons found on videos in the flat of one of the September 11 bombers. He has challenged and thwarted every attempt by the Government over the last decade to extradite him.
Three Siac judges, including Justice Mitting, ruled evidence from his former co-defendants Abu Hawsher and al Hamasher, said to have been obtained by torture, could be used against him in a retrial.
But Robin Tam, QC, for the Home Office, told the court it had ruled on a "possibility of a possibility".
Shadow Home Secretary Yvette Cooper backed plans to appeal the "extremely serious and worrying judgment", but said Mrs May had to get Qatada's deportation back on track.
Qatada's solicitor Gareth Peirce said: "It is important to reaffirm this country's position that we abhor the use of torture and a case that was predicated upon evidence from witnesses who have been tortured is rejected – rejected by the courts of this country as by the European Court."