Music and mayhem and murder with ex-indie A&R man John Niven

We are upstairs in Quo Vadis, a private club in Soho, London, taking tea in one of the warren of beautifully appointed rooms in the building where Karl Marx lived for five years and where he wrote much of Das Kapital. It's an appropriate setting given the Scottish author spent 10 years in one of the most cynically exploitative of capitalist industries, working as an A&R man in London throughout the 1990s, spanning the rise and fall of Britpop, New Labour's Cool Britannia and the Spice Girls.

Niven turned down the chance to sign both Coldplay and Muse, and has jokingly described himself as the "Dick Rowe of the noughties", a reference to the Decca Records executive who knocked back The Beatles. "I was only interested in signing things I really liked instead of ones that would sell," Niven says, "but looking back I'm quietly proud I signed Mogwai instead."

Niven channelled his experiences into Kill Your Friends, a scabrous, hilarious music biz satire told from the perspective of sociopathic music executive Steven Stelfox, a creation of extreme hideousness in the vein of American Psycho's Patrick Bateman. The book is set in 1997, the year Oasis's bloated Be Here Now sounded the death knell of the Britpop era.

Deemed a "hard sell" by Niven's agent, the manuscript was repeatedly rejected before Random House picked it up. Their faith was repaid when the novel became a word-of-mouth hit, garnering acclaim ("the best British novel since Trainspotting"), notoriety and an army of fans.

Today, more than four years since the publication of his career-spawning novel, and despite a literary reputation for pyrotechnical swearing, Niven is exceedingly polite, if honest and opinionated. Perhaps it is the genteel surroundings, for which his bespoke Savile Row houndstooth check suit is perfectly apposite. He has a dinner date tonight at exclusive seafood restaurant Scott's of Mayfair with a friend from his A&R days, Caitlin Moran, the broadcaster and writer of the recent bestseller How To Be A Woman. At 46 and prone to outbursts of laughter, he possesses the mien of a rich, benevolent uncle.

Niven peppers the conversation with quotes from famous writers and even manages to laud the Updikean qualities of Twitter without sounding pretentious.

"Twitter is almost novelistic," he begins, speaking with the authority of a 50-tweets-a-day man (his contention that it's not an addiction is difficult to swallow), "like what John Updike said about 'giving the mundane it's beautiful due'. You just get the essence of the mind in a very pithy sentence."

Maybe it's because Niven says it in an Ayrshire brogue that has lost none of its rough edges despite 20 years living in London and Buckinghamshire.

He shows me a photo on his phone of his garden shed cum office, his favourite place in the world. "I love a quality shed," he says. Carpeted, heated and lined with bookshelves, his writing lair has more in common with the elegant, wood-panelled library in a private members club than something you could buy in B&Q. As they might say back in his hometown of Irvine, in what he calls "the Ayrshire rust belt", the boy done good.

He breaks off to say hello to a passing publisher acquaintance, returning to tell me about his latest book, Cold Hands, his fifth novel and his first foray into the crime thriller genre. Cold Hands is a daring departure in style from his 2005 debut, Music From Big Pink, a novella based on The Band's classic 1968 album, and the subsequent trio of scandalously funny black comedies, Kill Your Friends, The Amateurs and The Second Coming.

I tell him I raced through Cold Hands, especially its bloody denouement, gripped by the creeping tension but desperate to reach the end and extricate myself from its nightmarish hold. Where does this apparently genial man, I ask, get such appallingly dark ideas?

"I'm afraid that's a whole bag of therapy I don't want to go near," he replies.

If he had one, Niven's therapist would have a field day analysing Cold Hands, a taut, violent, nerve-shredding page-turner of crime and punishment, rehabilitation and revenge.

"I knew from early on that there was no place for humour in this one," he says. "The idea was always going to dictate the tone and that meant there would be no comedy.

"In my other books the impulse was always to make readers laugh but here it was to keep them on the edge of their seat."

The book centres on the riveting, if uncomfortable, story of Donnie Miller, a house-husband living in the frozen hinterland of Saskatchewan, Canada, with Sammy, his powerful, wealthy wife and their eight-year-old son, Walt. Donnie whiles away his days writing movie reviews and an entertainment column for a local newspaper edited by his wife but harbours grander ambitions to be a screenwriter.

The book's narrator, Donnie is tormented by a terrible event in his troubled Scottish upbringing – described in vivid flashbacks – and when the family dog goes missing there is a dreadful foreboding that Donnie's past is about to come back to haunt him.

Niven's formative reading experiences featured novelists using the device of the writer as character – like Jack Torrance and Paul Sheldon in Stephen King's The Shining and Misery, and TS Garp in John Irving's The World According To Garp – which influenced him to make his protagonist a writer.

"It opens up certain narrative possibilities," he explains. "They have a licence to notice stuff, to shape it and present it in a particular way." In Donnie's case, his narration becomes ever more unreliable as the tension ratchets up to breaking point.

With the new direction comes a rebranding of the author as John J Niven. Why the need for the extra initial? "My publisher suggested adding it to make this one a little distinct from the last three," he says. "I was nervous because it can look pretentious but they didn't force me into it and hopefully most people will still know it's me."

With the outlines of the next three novels already mapped out ("although they'll probably bear little relation to the finished books"), he makes what Norman Mailer called the "spooky art" of writing fiction look easy. Considering he once said he had "let his brain atrophy for 10 years in the music industry", is he trying to make up for lost time?

"A wee bit," he reflects, sighing. "I feel like I've been given this last-ditch opportunity and I don't want them to take it away from me, and if I just keep working, writing all the time, then I'll be OK."

Another motivation, he half-jokes, is "tramp fear", the notion put forward by John Self in one of Niven's favourite novels, Martin Amis's Money, that only he can put money between himself and jail. "There's no family pot of gold to fall back on," he says.

John Jeffrey Niven was born in Irvine on May 1, 1966, the year the old fishing port was designated a new town. He was the first of three children born to Jeanette and John Sr, an electrician by trade who worked his way up to become the manager of Rivergate, the town's main shopping mall. Among the pebble-dash council housing estates of 1970s Irvine, Niven daydreamed about following in the footsteps of his musical, not literary, heroes.

He says he was "not a particularly popular kid" and never seemed to fit in at secondary school. He remembers his first few years at Ravenspark Academy as being an isolated and lonely period. At break time, an air of danger pervaded the playgrounds and corridors while in the classrooms, Niven remembers, "6ft-tall, fully grown men would clatter 11-year-old children with a big leather belt".

"It was just a terrifying, barbaric ordeal," he says, still feeling the sting of the taws. "It was a brutal world and the brutality came down from the top."

Anyone – especially male – who attended a comprehensive school in Scotland in the late 1970s or early 1980s will recognise this world as it's rendered with chilling accuracy by Niven in Cold Hands, when the focus shifts from Donnie's rural Canadian idyll to his harrowing schooldays at a Scottish comprehensive eerily similar to the writer's own.

A few months after his 16th birthday, while watching The Clash play the Magnum Leisure Centre in Irvine, Niven caught a glimpse of another reality. "The Clash were the first big love of my life," he tells me. "Lyrically they inspired me to get out, explore life and maybe kick some doors down." He taught himself guitar, started playing music with new friends and the world began to open up for him.

"As much as I slag off Irvine, there was a good little creative scene in the mid 1980s. We were all in bands and Andy O'Hagan [a friend from nearby Kilwinning and now a prominent novelist] even wrote a fanzine," he says, laughing.

A "wonderful" English teacher at Ravenspark, Jean Doole, sparked Niven's interest in literature (with Irving's Garp in particular leaving an indelible impression) and he was spurred on to become one of only three pupils in his year – and the first in his family – to make it to university, in his case the cloisters of Glasgow.

After two years studying English literature, during which he spent most of his time hanging out in Glasgow's thriving music scene, centring on the Splash One club, Niven joined indie hopefuls The Wishing Stones on guitar and in 1987 took a sabbatical from academia. The bright lights of London were calling but success eluded the band and after an acrimonious split in 1989, Niven high-tailed it north to finish his studies.

On leaving university with a first-class honours degree in English literature, Niven "stumbled into a well-paid, glamorous job" at London Records before spells at Independiente and PIAS. "It was great fun for a few years," he says, "but I was pretty burned out by the end."

Now he sees with the benefit of hindsight that his A&R career dragged on five years too long, but he was never good at leaving a party early, and, hey, they were all young and carefree and there was still a lot of money flying around the music business. They were hedonistic times. "That combination of youth, money and a lack of real responsibility – it makes for interesting situations, doesn't it?" he says.

I have to ask the obvious question: do Niven's own experiences bear comparison with the debauchery he describes in Kill Your Friends? "Well, one of the mechanisms a comic novel works by is exaggeration," he says, "but it's not that exaggerated."

Frazzled after a decade of partying, Niven was offered redundancy in 2002. Despite a fear of leaving his high-profile job, he took a leap of faith and also heeded advice from friends in the music industry, especially Manic Street Preachers singer and guitarist James Dean Bradfield, to pursue his talent for writing. "I'd always wanted to write but I had none of the self-discipline you need to write a novel," Niven says.

Around the same time he remembers receiving Stephen King's memoir On Writing as a birthday present. It had a profound effect. Niven recites a quote from the book about how writing fiction is a difficult, lonely job, "like crossing the Atlantic Ocean in a bathtub". It was time to embark on his own long trip across the pond. "I arrogantly figured I'd have enough money to live for a year and that would be enough time to write the great British novel."

In reality, it took three more years of working "feverishly hard", a little longer than he'd envisaged. And the result of "shutting my door for six hours a day for a full year" wasn't a great British novel but instead a delicate yet powerful mix of fact and fiction, Music From Big Pink, about a small-time drug dealer and musician manque on the fringes of The Band's circle as they compose their debut album in the house of the title near Woodstock.

Music From Big Pink was published in 2006 with Greil Marcus, the doyen of rock critics, hailing it "an amazing piece of work". A glowing review in the New York Times won Niven a top London agent and the follow-up, Kill Your Friends, confirmed his arrival. He was an "overnight success", some six years in the making.

Nowadays John Niven is busy juggling several writing projects and spent four months in Los Angeles over the summer, swapping his garden shed for a Hollywood house with a view of palm trees and a glistening outdoor hot tub, working on the script for a horror film with the makers of the HBO TV series Entourage.

His US management were keen for him to hang out in Tinseltown, go to meetings, develop relationships, "all the stuff that goes on in LA".

"Also, I knew a good chunk of the novel I'm working on now [a return to plain old John Niven, about a morally and creatively bankrupt novelist and screenwriter] was going to be set in Los Angeles, so I wanted to spend some real time there."

His only original screenplay to make it on to the big screen so far was, he says, an absolute disaster. The script he wrote with writing partner Nick Ball (the brother of broadcaster Zoe), called Roadkill, was sold to a Hollywood studio for an "obscene" sum and turned into the $13 million action thriller, Cat Run, a mess of a film, Niven says. At the glitzy premiere he barely recognised it as his own work.

"It was a crazy experience – every Hollywood cliche you could imagine," he says, laughing. "Thankfully nobody saw it."

He is developing an idea for a TV series for Sky Atlantic with fellow Scot Andrew Macdonald (producer of Trainspotting, The Last King Of Scotland and many others), an ambitious family saga covering four decades, spanning the birth and death of the music industry, from Beatlemania in 1963 to 1999 and the advent of Napster.

The film rights for all his books have now been sold, with Kill Your Friends – a low-budget British production scripted by Niven himself (with, he hints, a big-name actor to play his vile anti-hero) – the first to start shooting in January 2013.

Niven says there has been no pressure from the various film companies to tone down his colourful lexicon of Scottish profanities for a worldwide audience.

The Amateurs, Niven's third and arguably funniest novel, a golf comedy about a hopeless hacker transformed into a Tourette's-blighted Open contender after being hit on the head by a wayward drive, was his first to be set in Scotland and his favourite book to write. Amid the comedy carnage it is an oddly moving homage to the Ayrshire summers of Niven's youth when it seemed every boy was out on the golf course. It felt, he says, like coming home.

"I love writing Scottish dialogue," Niven enthuses. "It was natural to give Donnie in Cold Hands a Scottish past because I knew the way around, I knew what the furniture looked like, how he would talk. I'm outlining a thriller now and was thinking about setting it in the north-east of England, but I find myself leaning towards Scotland again."

Niven has a settled life in High Wycombe with his fiancee Helen, a lawyer, and their four-year-old daughter, Lila. "There's not a day I don't go down to the bottom of the garden and just about kiss the floor of my office," he says, "and think how incredibly fortunate I am that it all panned out."

His 16-year-old son Robin lives about half an hour away with his mother, Niven's ex-wife. The teenager recently beat his father at golf for the first time ("a sore day").

Though he remains exiled by choice, he toys with the idea of buying a flat in his old stomping ground. "I often find myself clicking on Hillhead estate agents' websites, wondering what I could get for certain prices on certain streets. And I even get the odd benevolent pang towards Ayrshire."

And with that he walks me out on to Dean Street, and Soho's Friday night hubbub, in search of a quick snack before setting off later for his posh night out. He could easily order food from the bar at Quo Vadis yet plumps instead to disappear into a nearby Tesco, in his Savile Row suit, for a pack of takeaway sandwiches.

You can take the boy out of Irvine -

Cold Hands by John J Niven is published by William Heinemann, priced £12.99.