From successive governors of Barlinnie Prison in Glasgow, who remain concerned about the number of inmates with mental health problems in that most notorious Scottish jail, to Government-appointed prison chief inspectors, to frontline prison officers, there is a surprising range of people who hold that view.
Andrew Coyle, a former governor of multiple prisons and now a respected academic and consultant on penal policy, is among their number. "The use of imprisonment in Scotland is disturbingly consistently high," he says.
This comes despite regular reports and policy changes recommending a change of direction. In relation to women offenders, for instance, reports by former First Minister Henry McLeish and former Lord Advocate Elish Angiolini have both called for reductions in the numbers sent to prison. But despite reductions in short-term sentences, the jail population remains stubbornly elevated, while Professor Coyle is alarmed by recent policy decisions relating to the prison estate.
"The Government accepted the Angiolini report, and said it would set up small units. Its version of that is 100 places within Edinburgh Prison, or relatively large units in new prisons like Grampian, " he says.
Such plans, and those for a new 500-place women's prison are not in line with successive reports, he adds.
New prisons at Low Moss, Bishopbriggs, and Addiewell, West Lothian, will be joined by a new one in Grampian next year, he points out. "These are big prisons in a small country like Scotland. Low Moss, Addiewell and Grampian are all high security prisons, which are very expensive."
It is such critiques that make it no surprise Mr Coyle has just been appointed the first president of the Howard League in Scotland (HLS). The penal reform charity's English counterpart has long had such a post, but Mr Coyle is the first for HLS.
However, he is far from a new convert to prison reform. His 1991 book Inside: Rethinking Scotland's Prisons was founded on successive spells as governor of Greenock, Shotts and Peterhead prisons.
It was influential in the modernisation that has taken place in Scotland's prisons, with more emphasis on rehabilitation, maintaining family links for those in jail and a transformation of the role and expectations of prison officers.
The change that has taken place over time is likely to be a focus in an inaugural lecture to be given by Mr Coyle next month. It is not the first time he has given the annual Drummond Hunter, which takes place this year on November 18. "I gave the second one in 2003, so I will be coming back exactly a decade later, analysing the things that have happened over the period," he says.
He intends to be far more than a figurehead for the HLS, and hopes to help the organisation pass on the message that prisons can't be seen in isolation and offenders likewise."The objective is to make Scots feel safer. People who break the law certainly need to face the consequences of their actions, but should not be defined by that action," he says.
"An offender may be a father, brother, mother, son. If that person is to contribute to society, rather than threatening it, our response needs to be much wider than focusing on what is rather quaintly called 'offending behaviour'."
HLS brings together in its membership many people who have been part of various criminal justice agencies and are well placed to contribute to policy at a strategic level, he says.
Meanwhile, Scotland has been in many ways at the vanguard of new approaches, he says.
"The work of Glasgow's Violence Reduction Unit is world famous. I don't think we always realise how cutting edge that work is.
"But the structure of community supervision and support needs to be reviewed. Criminal Justice Agencies were introduced a few years ago but after a good start, they have not developed."
Scotland has become a safer place, with crime overall falling, he says. "People feel quite safe in general terms, but there are places where people do not feel safe, and if you are in a part of Edinburgh or Glasgow that does not feel safe, that does not help you.
"Our criminal justice system still focuses on the activities of individuals, whereas the cutting edge thinking is saying that as well as individuals you need to look at location."
Politically, it has been difficult to move resources to spend more on tackling problems in the backgrounds of those who offend, rather than on simply locking people up, Mr Coyle explains.
He is also sceptical about the suggestions that a prison such as the new HMP Grampian, Peterhead, will be able to work closely with their local community, given their size.
"Because of the size, many of the prisoners will not come from the north east. It is difficult to see how a prison like Grampian can be community-facing. The last time I was at Peterhead, it was North Sea facing," he says.
"What encouraged me to accept the invitation by the HLS was that there is an opportunity to take things forward on a strategic front. A body like HLS, which has no axe to grind, is quite well placed to facilitate that work."
While the Howard League has largely been associated with prison reform, Mr Coyle is also keen to speak out on other justice issues.
He is cautious about the reaction to the Carloway Report, which called for an end to Scotland's laws of corroboration. Subsequent suggestions that the author was out of touch are wide of the mark, he says.
"People saying that have not read Carloway's Report, I suspect. Only Scotland has this system, but there are other ways of providing corroboration. It is actually quite a nuanced report."
Mr Coyle helped shape the new Scottish prison visiting system, which will see jail visiting move under the auspices of the Chief Inspector Of Prisons, although he says it will be important to keep the distinction between continuing monitoring and less regular inspection.
John Scott, chairman of HLS, says he is delighted the professor has agreed to take on the task of giving the charity a higher profile.
"Andrew has obviously done exceptional work in the field and we have worked with him before," he says.
"He will be able to help us in variety of ways, and has his own ideas about penal reform and other improvements in criminal justice services, not to mention a wealth of contacts and experience.
"He did the job and learned from it and used the experience to suggest better ways of doing things."
On the back of campaigns on prisoner voting and other issues, HLS has increased its membership and influence, Mr Scott argues.
But he shares Mr Coyle's concerns about the current prison building programme.
"Increasing capacity is a surefire way of making sure the prison population does not diminish.
"We would want to see people held in humane conditions, but if you build new prisons you are going to have to fill them."