The taxman has often faced accusations of imposing draconian fines for people who can't get their returns in on time.

But it has emerged HM Revenue and Customs is so tardy at answering the phone that it has cost the taxpayer £136 million a year.

The National Audit Office (NAO) today branded the collector "totally unacceptable" after 20 million calls to its hotlines – many of which are costly 0845 numbers – were not picked up at all.

People who did get through were also waiting more than twice as long to speak to an adviser; an average of 282 seconds compared with 107 seconds in 2009/10.

In the first quarter of this year, 6.5 million people were left holding on for longer than 10 minutes.

The NAO found there had been some progress after thousands more staff were drafted in with the 74% pick-up rate significantly higher than the 48% recorded in 2010/11.

Yet the report warned the figures probably underestimated the issue as calls are counted as answered even if they do not reach an adviser.

Labour's Margaret Hodge, who chairs the Committee of Public Accounts, said: "I find it totally unacceptable that HMRC uses costly 0845 numbers and charges people for the privilege of waiting for the department to pick up."

Sluggish performance amounted to a taxpayer telephone toll of £136 million in lost customer time and call charges in 2011/12. She said current call answering targets were "far too soft".

The NAO estimated that in 2011/12, it cost customers £33 million in call charges while they waited for HMRC to answer the phone and the estimated value of customer time while they waited was £103m.

Catherine McKinnell, the Shadow Treasury Minister, urged ministers to "get a grip".