Giving evidence to MPs, Mark Sedwill said there were no surviving records of what had been destroyed or lost from the 1980s and 1990s.
He also announced Richard Whittam QC will be leading a fresh inquiry into the Home Office's handling of information it received during the period - including a 1983 dossier from ex-MP Geoffrey Dickens.
Mr Sedwill told the Home Affairs Select Committee: "I am concerned about all the material that we cannot find."
The permanent secretary's comments came after home secretary Theresa May revealed former High Court judge Baroness Butler-Sloss is to lead a wider independent inquiry into how claims of child abuse were dealt with by public institutions, political parties, the church and the BBC.
Rumours of decades of organised paedophile activity among the ruling class have moved centre stage over the past week, amid questions over whether the Home Office failed to act on detailed allegations provided by Mr Dickens in 1983.
Then-home secretary Lord Brittan has flatly denied failing to deal with the material properly, while a review carried out by a HMRC official last year found no evidence anything relevant was not passed to other authorities.
However, it also disclosed the Dickens dossier appeared to have been destroyed - and Mr Sedwill has since disclosed 114 files deemed potentially relevant are missing.
Mr Sedwill said he decided to draft in an expert investigator to look into the Home Office's handling of paedophile allegations last year in response to questions from Labour MP Tom Watson.
He stressed the inquiry had not found any evidence documents had been inappropriately destroyed.
But the permanent secretary added: "I am concerned about all the material we cannot find."