He was once the enfant terrible – or should that be wicked wean – of Scottish letters.

Iain Banks's first novel was breathtaking, audacious and more than slightly unsettling. The Wasp Factory was published in 1984 and Banks, now a late fiftysomething, can peer back at a body of work that is substantial in output, perennially popular and regularly diverting.

This is his 26th novel, though 12 are published under Iain M Banks, his nom de plume for science fiction works. Stonemouth is a long way from the The Wasp Factory and crucially much nearer to his most successful and more orthodox novels such as The Crow Road and Espedair Street.

This ensures that Stonemouth has its strengths, but it also means that its familiarity in tone and style will not excite. Banks works in the gap between high art and low-level but commercial schlock. Both these genres make demands of their writers but Banks walks the tightrope of trying to produce a book for a mass audience while imbuing it with some sort of substance. This high-wire act is most notably performed by such as William Boyd and Robert Harris, but it is dangerous work. Apart from a focused professionalism, there must be an element of risk.

It is the lack of the latter that is the greatest disappointment in Stonemouth. Banks has a practised ease in storytelling, he can create a central character who is both engaging and ever so slightly flawed, and he produces dialogue that is funny and convincing.

The narrative breezes through a town on the northeast coast of Scotland. It begins at a tilt with a clever scene on a suspension bridge that produces, well, suspense. It becomes immediately clear that the hero has had to leave his home town suddenly some five years before and his return holds great danger. He has upset the local gangster family, led by Don Murston rather than Don Corleone.

Banks is technically adept at unfolding the story over a matter of days while harking back to the childhood, adolescence and great love of his narrator and hero. There are some wonderful scenes with nods to Golding and Frost. The horror of a trip into the woods by a group of boys recalls The Lord Of The Flies while the narrator's preference for solitary walks evokes The Road Less Travelled.

Stewart Gilmour – hero, narrator and lovelorn exile – is made to carry the story, though, and he sometimes stumbles under the weight of what is, ironically, a lightweight prop.

On the surface, this is the story of a love gone wrong and the guilt, regret and anger that break-up causes to a couple and to a frankly psychotic family. It does not convince.

The plot twist is more of a three-point turn performed when Banks is heading for a cul-de-sac and instead seeks to point to open road with a variety of possibilities. It is clumsy, particularly when set aside the smoothness of the rest of the novel.

Billed as both a thriller and a love story, Stonemouth comes up short on both. There are moments of menace, particularly when the gangster family is evoked rather than engaged. The Murstons work better as a threat by sitting in the shade of the background rather than being brought into the light, where their evil becomes a caricature rather than something more frightening. Banks, too, runs out of pace at times. Raymond Chandler once attested that if he was blocked with a plot he would just have a man with a gun bursting into a room. But there are only so many times that a demented Murston can be relied upon to produce some drama.

The love story, too, has its flaws, though it testifies to what is best in the novel. Elie Murston is a captivating character. She is mysterious, possessed of a Caledonian melancholy, and trapped in both a town and in a family that causes her debilitating unhappiness. Stewart Gilmour could and should have been her way out. The failure of Stewart to perform this duty haunts him but Elie remains constantly intriguing, somewhere just beyond the ken of the reader.

It is in these moments that Banks shows his prowess as a novelist. However, there is a distinct feeling about halfway through Stonemouth that the thriller element of the novel cannot engage its creator. This theory is reinforced by the crude way this is resolved. There is subtlety in other areas, particularly in the dynamic between Stewart and Elie. There are also hints that Banks could have taken other roads and, indeed, explored those more deeply.

Stewart is certainly an exile but he is impelled to come home. Elie is betrayed but she has a need to trust. Banks has called up two compelling characters from the haar of a Scottish fishing town, but they slip back into the mist leaving only a sense of frustration behind. This feeling is exacerbated by the knowledge that Banks has the ability to intrigue and has a mastery of form and a gift for dialogue.

Stonemouth starts with Stewart swaying on a bridge as waves crash far below him. It is essential to the plot that he walks away from that scene and becomes the hero of a standard Banks novel. Stonemouth, though, would have been much improved if the author had made a leap of faith into uncharted waters.

"I look down at the waves again, wondering what Callum's last thoughts were as he fell towards the water and whether he died without waking up and whether he had time to suffer. I suppose every class at every school has a first person to die – suicide, road crash whatever – just like there's a first person to get pregnant or father a child and a first person or a first couple to get married. Callum was not our first death but he was our first suicide. Our first death was Wee Malky, long ago. Well not just our first death; something worse in a way ..."