The Prime Minister's official spokesman said the decision to stand down was entirely Lady Butler-Sloss's.
Her departure comes after the former judge's appointment became the centre of controversy because her brother Sir Michael Havers, who was attorney general and lord chancellor in the 1980s, is alleged to have tried to prevent ex-MP Geoffrey Dickens airing claims about child abuse in Parliament.
Critics have suggested that victims of child sex abuse would not have confidence in an inquiry led by someone from the heart of the establishment.
It is understood that Lady Butler-Sloss informed the Home Office of her decision to step down over the weekend, after which she spoke to Home Secretary Theresa May.
The process of selecting a replacement is beginning immediately, but a new name is not expected to be announced for some days.
David Cameron's spokesman said: "She has taken the decision to step down as chair of the panel inquiry. It is entirely her decision.
"The Government's view hasn't changed, that she would have done a first-class job as chair. The reasons for her appointment still absolutely stand in terms of her professional expertise and her integrity, which I don't think has been questioned from any quarter whatsoever, and rightly so."
In a statement, Lady Butler-Sloss acknowledged that she "did not sufficiently consider" whether her family links would cause difficulties in the inquiry.
Lady Butler-Sloss said: "I was honoured to be invited by the Home Secretary to chair the wide-ranging inquiry about child sexual abuse and hoped I could make a useful contribution.
"It has become apparent over the last few days, however, that there is a widespread perception, particularly among victim and survivor groups, that I am not the right person to chair the inquiry. It has also become clear to me that I did not sufficiently consider whether my background and the fact my brother had been attorney general would cause difficulties.
"This is a victim-orientated inquiry and those who wish to be heard must have confidence that the members of the panel will pay proper regard to their concerns and give appropriate advice to Government.
"Nor should media attention be allowed to be diverted from the extremely important issues at stake, namely whether enough has been done to protect children from sexual abuse and hold to account those who commit these appalling crimes.
"Having listened to the concerns of victim and survivor groups and the criticisms of MPs and the media, I have come to the conclusion that I should not chair this inquiry and have so informed the Home Secretary.
"I should like to add that I have dedicated my life to public service, to the pursuit of justice and to protecting the rights of children and families and I wish the inquiry success in its important work."
Mrs May responded: "I am deeply saddened by Baroness Butler-Sloss's decision to withdraw but understand and respect her reasons. Baroness Butler-Sloss is a woman of the highest integrity and compassion and continues to have an enormous contribution to make to public life.
"As she has said herself, the work of this inquiry is more important than any individual and an announcement will be made on who will take over the chairmanship and membership of the panel as soon as possible so this important work can move forward."
Lady Butler-Sloss was appointed on July 8 to head an inquiry into allegations that public bodies, including Parliament, failed to protect children from abuse over a number of decades. A separate inquiry under NSPCC chief Peter Wanless is also reviewing the Home Office's handling of allegations between 1979 and 1999, including a dossier submitted by Mr Dickens in 1983.
But the choice was questioned the same day by Commons Home Affairs Committee chair Keith Vaz, who said he was "surprised that the Government has chosen a member of the House of Lords, no matter how distinguished, to head the inquiry".
Mr Vaz said: "She is a member of Parliament and is very closely related to a former lord chancellor."
Pressure mounted at the weekend when she was reported to have told a victim of alleged abuse she did not want to include the allegations in a review of how the Church of England dealt with two paedophile priests because she "cared about the Church" and "the press would love a bishop".
Former solicitor general Vera Baird said today the appointment was "an error" because the former judge's family connections meant she had a conflict of interest.
Mrs May was due to be questioned about the appointment when she appears before the Home Affairs Committee in Parliament this afternoon.
Asked whether the Government would now be looking for an inquiry chair who was not identified so closely with the establishment, Mr Cameron's spokesman told a Westminster media briefing: "The key thing around appointments will be getting a panel that has the right range of skills and expertise and credibility that gives and inspires confidence in it and its work.
"The view has not changed, for the Prime Minister and Home Secretary, that Lady Butler-Sloss could have done that."
The spokesman declined to say whether Lady Butler-Sloss's family connections were considered in the process of selecting her as inquiry chief, saying only: "I don't think it was a particularly public secret.
"The Government looked at the full range of ways in which she was a highly credible person to take up this role. The key points were her expertise and the ground-breaking work she did around the Cleveland inquiry."
Discussions on the make-up of the panel, including legal and child protection experts, who will conduct the inquiry are not expected to begin until a new chair is appointed, in order to allow the new inquiry head to take part in the process.
Mr Vaz, who raised questions over the appointment with the Home Office's most senior civil servant, permanent secretary Mark Sedwill, said the inquiry process was becoming "shambolic".
He said: "I am not surprised by this decision - it is the right one.
"As I pointed out to Mr Sedwill, the public would be concerned that a member of Parliament, no matter how distinguished, had been appointed to head this important panel.
"The whole inquiry process is becoming shambolic: missing files, ministers refusing to read reports and now the chair resigning before the inquiry has even commenced."