But he insisted he was opposed to decriminalisation and that the Government had no intention of relaxing the law.

Mr Clarke delivered his gloomy assessment when asked at the Home Affairs Select Committee what his long ministerial experience of dealing with the issue had taught him.

"I have not reached the stage of that blinding insight about how we are going to improve our record, is the honest truth," he told the cross-party panel.

"We have been engaged in a war against drugs for 30 years. We're plainly losing it. We have not achieved much progress. The same problems come round and round. But I do not despair – we keep trying every method we can to get on top of what's one of the worst social problems for the country and the biggest single cause of crime."

Pressed on whether decriminalisation could be a solution, he said: "The Government has no intention whatever of changing the criminal law on drugs. I have frankly conceded that policy has not been working. We are disappointed that far from making progress it could be argued we are going backwards at times. But my own personal view is that I would be worried about losing the deterrent effect of criminalisation of youngsters who start experimenting.

"The key thing is to try to work out how to get fewer young people to start experimenting with drugs. One thing that does put them off is that they'd get into trouble with the police."

He said he did believe one area of improvement was that "friction" between Whitehall departments over whose responsibility it was had lessened.

It "has not vanished but is very much less than it used to be", he said, praising the coordination between his department and the Department of Health.

Mr Clarke also vowed to investigate after prison chiefs told the committee they had not changed policy on methadone, despite a new Government strategy.

He expressed surprise at evidence given by Richard Bradshaw, of the National Offender Management Service, that its treatment of heroin addicts remained unchanged.

He said: "I think Andrew Lansley and myself, Nice and the prison service had better touch base afterwards about whether we are or are not moving towards a treatment system focused on recovery."