Like any good Glaswegian, Peter Mullan can make every word sound threatening, even words like "kitten", "fluffy" and "cuddle".

"I want a cuddle," he said in the first episode of his new drama The Fear (Monday-Thursday, Channel 4, 10pm) and never has the phrase sounded so terrifying. But English viewers, do not panic: a Glasgow cuddle is nothing like a Glasgow kiss.

The problem for Mullan is that this kind of dark, terrifying role seems to have become a bit of a trap for him. Because of the way he sounds (apart from Sean Connery, he's the only actor allowed to have a Scottish accent in every role) and the constant what-you-lookin-at-pal expression on his face, he tends to play the same part on repeat: the hard Scotsman with a hard life.

At first glance, his character in The Fear, Richie Beckett, was the same type: a former gangster who has punched and kicked his way to dominance of his patch in Brighton. But what made the character much more interesting was the fact there were cracks in the gangster carapace, little faultlines appearing in the toughness that suggested he was losing control.

The problem, it emerged, was dementia and what a fascinating, original idea that is: to tackle what happens to a man with dementia if that man is a violent criminal. This is someone who has probably told a thousand victims that he will never forget their face and now here he is doing exactly that.

Mullan played the effects beautifully because he is a strange and great combination: he can play the swagger and animalistic confidence of a man who's had a hard upbringing, but always he has subtlety and pathos as well. He can snarl and clench his fist and yet make us ache with sympathy for him.

This is exactly what he did in The Fear, aided by some experimental direction by Michael Samuels, designed to make us feel just as confused as Richie felt as the dementia crept deeper into his brain. What was particularly witty was using violence to make the point. Early in the first episode, Richie beats up a man in the street but later he acts as if nothing has happened. At the police station, suddenly the pat phrase of the criminal refusing to cooperate – "I have no memory of the incident, officer" – means something.

Using a violent storyline in this way made The Fear a little queasy and uncomfortable in places (the scene with the dismembered girl in the bed went too far) but it also made it more exciting than a conventional drama about dementia, set in an old folks' home or a hospital, could have been. For Richie, the condition meant not being able to stay in control of his criminal empire; it meant one nasty violent man would be usurped by other nasty violent men.

But a controlled script and inventive direction kept the discomfort and compulsion in perfect balance. The performance from Mullan, with so much weariness hanging on his face and all that violence and fear in his eyes, was a disorientating triumph and the drama was a nerve-wracking experiment: grimly dark, but frightening too, and marvellous and shocking.