Imagine the scene on your street.

You live at No.8. A new neighbour moves into No.9. The man at No.12 says this new neighbour is a paedophile. A beast. You are frantic with worry. What about your children?

Sex offenders are the most likely criminals to be reconvicted, right? Not so. Scottish Government statistics show that a year after release, only 14% of sex offenders have been reconvicted, compared to more than 90% of those jailed for dishonesty offences. In addition, government reports suggest "a significant proportion" of sex offending is unreported which means your new neighbour is far from the only risk. (What do you actually know about the vocal man at No.12?)

But there's an even more unpalatable factor to consider. Statistically, another house on your street poses a very significant danger to your children: No.8. Your house. Most abuse against children is committed by family friends and acquaintances. More than a quarter is committed by a relative.

Last year, the Jimmy Savile case propelled sexual abuse into the public consciousness on a new scale, leading to the arrests of public figures such as Stuart Hall, Rolf Harris and Jimmy Tarbuck.

Savile was perfect tabloid fodder: the "monster" who was prolific, lacking in empathy, unstoppable. But have such headlines skewed our deeper understanding of abuse and if so, what are the facts? Who are these people and why do they offend? More importantly, can their behaviour be changed?

A quiet revolution is taking places in Scottish jails. Prisoners are no longer housed in a single centre at Peterhead, and a radical new rehabilitative programme called Good Lives, based on the most recent research, has been introduced.

The premise of Good Lives is that – at least sometimes – offenders can change. Good news given that the number of sex offenders has risen steadily in the past five years. That rise must be seen in context: Scotland's prison population as a whole is increasing and, at 8000, is proportionately among the biggest in western Europe. Sex offenders remain a steady 10% of the total. The majority are based at Glenochil, near Alloa, with the rest spread between Barlinnie, Edinburgh and Dumfries. They are almost exclusively male; prison chiefs says the number of female offenders is "so small as to be insignificant".

The historic nature of many abuse cases partly explains the rise in numbers. Public attitudes have changed and victims are coming forward because they have more confidence that they'll be believed. This means offenders are, on average, older than mainstream offenders, which creates huge challenges for the prison service.

"The biggest issue for us," says Nigel Ironside, governor of Glenochil, "is the long-term health care of old men: early on-set dementia, angina, heart disease, diabetes, cancer. What does palliative care look like in prison?"

Good Lives is a cognitive behavioural therapy programme that challenges thinking, examines reasons for actions, then encourages alternative ways of behaving. It is used with all types of sex offender: rapists, paedophiles and non-contact offenders such as exhibitionists, but it is important to acknowledge there is no "cure" for deviant behaviour. Some offenders will never change.

"There are people who are prolific, manipulative and want to keep doing it," admits Paul Cairney, psychology manager at Glenochil. "They are very dangerous, but they are not a massive group.

"Feedback from participants on the Good Lives programme has been positive. They seem to identify that the programme is not just about assessing risk but is balanced between managing risk and building a positive future which will lead to an offence-free life."

No group is more reviled than paedophiles. But the danger of extreme labels is that you forget how very ordinary such people can be – and where to look for them. It also means you remove the motivation for offenders to conform to societal rules: if you're a beast, why behave like a human being? For that reason, Good Lives emphasises warmth and empathy in its delivery style.

"These guys need to open up," explains Rachel Roper, head of psychological services for the Scottish Prison Service. "They need to understand why they have done what they have done. A lot of them are ashamed and desperately want help. If I asked you about your deepest, darkest secret, it would be hard for you to open up if you didn't trust me. If you are then criticised and ridiculed, it's not going to work."

Sexual offences account for only 2% of total crime but the label "sex offenders" suggests a unified group that doesn't exist – even among those who have committed similar offences. Not all paedophiles even have a sexual preference for children.

"There are two groups," explains Roper. "One, if you gave them a choice of sexual partner, would say their preference was a child. With them, it's a bit like saying, 'I want you to like oranges.' They say, 'But I like apples.' It's very difficult to change preference – but not impossible.

"The other group's preference is actually for adults. They can't achieve that, so they turn to a child. They may be inadequate, have poor social skills, or don't know how to communicate. They may be lonely and isolated. They develop an inappropriate interest in children because children are nice to them and adults aren't. It's far easier to treat that."

It might seem difficult to behave warmly to someone whose behaviour you abhor, but the ability of prison staff to form non-judgmental relationships is striking. Some offenders are likeable, others are not, admits Roper. But they have to believe they are capable of being liked. "We don't condone their offences, but we do tell them, 'You are not just a label. You are also a human being. A friend. A brother.' Some of them, nobody has said a nice word to them in their lives. They have to have self-belief. If society sees them as beasts, they feel worthless and there is no confidence they can change."

If that sounds too understanding for your taste, it can be viewed another way. "Our goal," says Roper, "is to protect the public and have fewer victims."

At Glenochil, Jerry ambles into the room, dark hair cut into the wood, eyes cast down. Hesitantly, we shake hands. He makes only momentary eye contact. Jerry is 41 and, as he says himself, "a big boy". In fact, he had a heart attack recently after winning a prison badminton competition. He tried to ignore the pain tightening in his chest, soaking three towels with sweat before seeking help. At hospital, where a stent was fitted, he was asked for his next of kin. "Glenochil prison," he said.

Jerry is one of the first to complete Good Lives. The 350 sex offenders at Glenochil are segregated from the 320 mainstream prisoners for their own safety. Sex offenders are bottom of the prison hierarchy – and even they have a pecking order. "I'm just a rapist. I wouldn't touch a child."

Jerry is a Schedule One offender. That means offences against a minor. That means lowest of the low. I want him to tell me about his offence but, first, I'd rather he told me about his life. Jerry nods, eyes still on the floor. Home was the east end of Glasgow. He had two older sisters and two younger brothers but was closest to his father. His dad kept pigeons and father and son looked after them, went everywhere together. But until prison, Jerry never told a soul that his dad sexually abused him from the age of five. When he was 11, it emerged that his two sisters were also victims. His mother threw his father out but Jerry was furious. She had destroyed everything. Taken away the one person who loved him.

Jerry ran wild, ending up in care until he was 17 and imprisoned at 20. He spent large chunks of the next two decades inside for assault, robbery – and now a sex offence. Through it all, he pushed people away, believing that if you showed vulnerability, people took advantage.

"I didn't need anybody. I needed me." But that was the old Jerry. "He was just an animal. He didn't care about people." And now? "I realise that for every one person who wants to criticise you, 10 want to support you. I do care about people. I want a true family. I want people to help me. If you have people in your life, you keep them there."

The effect of Good Lives on Jerry is evident even to a stranger: the man whose gaze is erratic chooses direct eye contact when describing his offence. He has been made to confront his behaviour so frequently in group sessions that he now talks with an almost disconcerting directness.

Jerry's life was a mess. He rushed into a marriage that collapsed, fell into depression, but felt unable to leave because he had a daughter and a stepson. And then? "I touched a nine-year-old, a family friend. I put my hand between her legs on two occasions." He can't explain why he chose a child. It didn't even seem sexual. "There was a bit of me that said 'this is an escape from this relationship' because that's what happened with my da. I didn't have the balls to walk out on my kids. The first time I did it, I had an image of my da, of the things he made me do, and I stopped."

He gets angry when likened to his father. What does he feel about him now? "Pure hatred."

He served half his four-year sentence before release, but was recalled after failing to comply with simple things his social worker asked him to do, such as turning up for appointments. That failure changed his life. "I was a bitter man. It was the social worker's fault. It was everybody's fault but mine." But one day, he sat in his Barlinnie cell looking in the mirror, and somehow saw himself as he really was. "I was talking to myself and said, 'It's naebody's fault but your ain. What if you had done this or not done that? The social worker only asked you to do that – why couldn't you do it?'"

He agreed to attend SOTP (Sex Offender Treatment Programme), the forerunner to Good Lives. SOTP was innovative in its day but was a one-size-fits-all course. Research suggests programmes such as Good Lives, which is more flexible and tailored to the needs of the individual, is more effective. It will be years before Good Lives can be formally evaluated but there is cautious optimism in the prison service.

It is a rolling programme, with eight to 10 offenders per group, and prisoners have to discuss and analyse their offences and behaviour – while helping others do the same. Without fixed entry and departure dates, the group constantly evolves, and the course takes around nine months to complete. For Jerry, the programme's emphasis on life history was critical.

"You start with your life story then do sessions about relationships, family, friends. You don't realise at the time but when you look back, all the sessions interlink. How you manage problems in your life – that one really got me. How did I manage problems? I avoided everything. I know it sounds weird but I'm glad I got recalled to prison. Good Lives has really done it for me. It's no use me saying I changed myself. I did have that lightbulb moment, but Good Lives opened my eyes to a bigger change. It has torn me apart – but in a good way."

Only one friend has stood by him, but Jerry doesn't blame those who haven't. It's his offence, not theirs. His friend told him he hated what Jerry had done and always would. But he didn't know that Jerry. He only knew kind Jerry, who always helped him when he was in trouble.

Other people's faith affects Jerry. At an assessment meeting where staff and social workers praised his achievements, the once violent, emotionless man started to cry. "I just wasn't used to it. All anybody ever said to me was, 'You're like your da. You're going to jail.'" He doesn't even know if his father is alive and doesn't want to. It's his relationship with his mother, who died in 2002, that consumes him. He once drove from Ayr to Glasgow because she had no bread and milk, yet they could never sit and talk because in his heart he blamed her for the collapse of his family.

"I caused her all the grief under the sun. She'd just found out her two daughters had been tampered with by her man and then you have a wee bugger like me smashing windows and bringing more people to her door. That was the last thing she needed. I didn't see it before but with Good Lives, you get to see yourself. You get to see that wee bastard. He started at 11 years old and not once did he change." Jerry's eyes fill with tears. "People say they wish they could turn the clock back. I wish that too. I don't want a better life. I want my mum."

He can never put things right with her. But social workers will help him trace family and maybe his sisters will give him a chance. What about his victim? "She will have a difficult life," he says sombrely. She might have psychological problems. She might be frightened of men. He caused that. "I wish I could go out and buy her everything she needs to make her a better person. But no amount of money can do that." Did he deserve his sentence? "No," he says. "I deserved more."

'What we do," says Nigel Ironside, "is not about getting rid of risk. It's about reducing risk. Risk management is always an educated assessment based on the information that is available. It's not if we get it wrong but when we get it wrong. I say that guardedly, but it is an absolute reality."

Some parts of the world – Scandinavia, for example – have different expectations with regard to offenders. "They have a cultural acceptance that there is a real need to reintegrate people into the community," says Ironside. "Here, tabloid scaremongering about sex offenders creates a difficult environment in which to try to shape public perception about giving people the opportunity to reintegrate."

Ironside and his colleagues believe in people's capacity to change. That's their experience. Yet as a society we spend three times as much locking people up as we do trying to stop them reoffending. A place on the Good Lives programme costs only £4000 per prisoner but there is still a waiting list of prisoners who want to participate. People such as Jerry can slip through the net. Or Dennis, another Schedule One offender.

Dennis presents a different aura to Jerry's subdued contrition. Upbeat. Confident. It seems strange hearing a sex offender say he likes himself. But it's vital Dennis does. Look closely and the razorblade scars criss-crossing his arms tell you he was a self harmer. His back and legs are covered too. "I used to hate myself." Dennis's new self-esteem means he is eager to prove he's worth something.

At first, his description of childhood sounds normal. But gradually it emerges that he was sexually abused by a male relative. Beaten continually with a belt by his father. Scapegoated within his family. Sex offenders' relationships are often dysfunctional because that is what's familiar: their experience says relationships are violent, inappropriate and strangely confused with love. "I have yet to meet a prisoner where I can't make a link back to their childhood," explains Chris Medhurst, lead facilitator for Good Lives at Glenochil. "Sometimes they can't make it but I can. It's like walking into the deepest jungle and saying to a tribe, 'See this cannibalism thing – it's not the way forward. And they think, 'But my mum eats people - my dad eats folk.'"

Dennis dealt with life by drinking. As an alcoholic, he says, he "didn't give a s*** about anyone". But he thinks the buried, caring part of him affected his employment choices: he worked as a nursing assistant, in care homes, in a hospice. Alcohol dulled his responses. He began watching internet pornography. "The more I watched it, the more I thought it was real and the more I wanted to experience it." Like Jerry, Dennis maintains eye contact when describing his offence. He groomed three girls between the ages of 12 and 16 over a four-year period. Why children? "It was easier," he admits.

It could seem a shockingly cold admission; really it's a sign of progress. Denial is common. It took a lot of work to admit to grooming and acknowledge why he chose minors. On Good Lives, he had to write a letter to his victims. It took him four days. "I was as low as I had ever been. I went through a phase of not knowing who I was. I knew who I had been. I knew who I wanted to be in the future. But this person who I was, I didn't understand him. The journey wasn't nice but when you come out the other side, it gets better and better, and you realise you are not the horrible person you thought you were."

Dennis is upbeat for a reason. He is being released next week.

Once, the prospect terrified him. Now he can't wait. He has a housing appointment on his release date. So if he moves into No.9, will you have a beast next door? That's what Jerry used to call sex offenders – before he became one. He hated them. He was the man at No.12, shouting about the new neighbour.

Dennis just wants a chance to prove he will never cause anyone hurt again. "I am glad I went into custody," he says. "It has helped me become the person I was brought into this world to be." n

The names of offenders have been changed.