Iam almost sitting on the lap of Jason Isaacs.

We're in his trailer on the set of BBC drama Case Histories. My peers among the press pack, quicker off the mark, have grabbed the only two pews across the tiny table leaving little choice but to wedge myself in next to him. I've tried to perch, ladylike, on the edge of the narrow bench, but the chivalrous actor is having none of it. "Sit down properly," he insists, gesturing to the postage stamp-sized space. It's like a world record attempt for the most people crammed in a caravan kitchen.

Speaking to him involves contorting my body at an unnatural angle, a feat made all the more difficult by the fact I've been cautioned not to look directly into his eyes. Not by Isaacs, I should add, but almost everyone I speak to in the preceding days. "Wait until you see those eyes," said one ominously. "They're like two mesmerising pools," added another so breathlessly I feared she may require smelling salts.

I sneak a peek. They are a deep, piercing blue. Almost - sorry, where was I? Put it this way: it's not difficult to see why his on-screen depiction of the dashing and troubled private investigator Jackson Brodie has garnered such an adoring fanbase.

Isaacs, eating a beef stir-fry for lunch, is taking care not to accidently spear me with an elbow, or worse, fork. But even casually wiping a dribble of sauce from his mouth with thumb and napkin, he smoulders.

The actor is in Edinburgh to film the second series of Case Histories, based on the quartet of books by Kate Atkinson. When the last episode was screened almost two years ago, the incorrigible Brodie finally seemed to be coming to terms with the childhood death of his sister.

He was resigned to the fact his young daughter and estranged wife were moving to the other side of the world. Romance was blossoming with DC Louise Munroe, played by Amanda Abbington. Or so it seemed.

In the intervening period Brodie has "f***ed it all up royally", says Isaacs with trademark bluntness. He's run off to New Zealand, gone completely off grid, and now Munroe is engaged to someone else. "It's not a coincidence he loves listening to country music," says Isaacs, wryly.

There may be murder and intrigue, but Case Histories, which returns for three 90-minutes episodes next weekend, isn't a crime drama in the usual sense. Rather, as Isaacs explains, "the stories walk this incredibly fine line between absurd and tragic". It's as much as about satire, anthropology and the human condition as it is tracking down an absconded parent or the killer of a prostitute.

"He doesn't walk around like Poirot at the end," says Isaacs. "There aren't a bunch of flashbacks where it turned out to be the man wearing the green sweater with the van that left tire tracks in the mud. The clues are always in people's make-up and their personal flaws."

Few bear more emotional scar tissue than Brodie. "He is everything women want in a man, including how horribly messed up he is," says Isaacs. "He's one of those people who women think they should be able to save. They don't really want him – if they actually came across someone like Jackson it would be a nightmare. He's a fantasy figure."

Which brings us neatly to Isaacs's own heart-throb status. The 49-year-old actor, who played Lucius Malfoy in the Harry Potter films and Detective Michael Britten, a man trapped between parallel realities in US sci-fi television series Awake, is a surprisingly reluctant sex symbol.

"That's bollocks," he scoffs, when the topic is raised. "I've done a million jobs between last time and this one – and none of them have been sex symbols.

"He's a great character. Me? I like to think I'm vaguely chameleon-like and can play anything from a five-year-old Chinese girl to a 200-year-old Buddhist monk. This part struck a chord with audiences – I'm just lucky enough to get to play him.

"I have played many a non-heroic figure and I'm sure I will again. I just played a hideous, evangelical fundamentalist with three wives, including a teenager who I raped, in a film in New Mexico. I doubt whether that is going to enhance my sex symbol status much."

But, I counter, what about the YouTube clips? There are copious homages, meticulously spliced together from his films and television shows, bearing simpering titles such as Jason Isaacs: Hot Mess.

"I know there is all this stuff on the internet, but everyone who has ever appeared in public has someone, somewhere, who is editing things together about them," he says. "You could spend your entire life googling yourself and looking at things people have written about you, but in my own life I'm happy if I'm not the subject of utter disdain from my children and wife.

"My friends are monumentally unimpressed by me. It's lovely that people like watching the work, but not for a split second do I take any of it seriously.

"For instance, yesterday I had a fight with a guy twice as big and a hundred times as strong as me. He could take me out if his legs and arms were tied behind him and both eyes gouged out with a spoon.

"In Case Histories, guess what? I win the fight. There is no part of me that thinks I'm tough, sexy, cool or funny. I'm just lucky I get to be all of those things in make-up at work. There is nobody putting me on any kind of pedestal in my real life."

The second youngest of four sons of Jewish parents, Isaacs grew up in Liverpool before the family moved to London when he was 11. While his brothers became a doctor, a lawyer and an accountant, Isaacs plumped for a career as a thespian.

He is married to documentary filmmaker Emma Hewitt, whom he began dating at drama school, and the couple have two daughters – Lily, 10, and seven-year-old Ruby. They are based in Los Angeles, which meant filming in Scotland last year proved a wrench. Indeed when it was confirmed the second series would begin shooting in September, as the new school year was about to start, Isaacs almost turned the part down.

"As much as it killed me and I wanted to do it, there was no way I was going to let my wife and kids go to America without me," he says. "But Emma, who takes not the slightest bit of interest in my career and is interested in me, correctly, as a husband who can take the rubbish out and kids to school, said: 'You've got to do it.'

"I said: 'Are you mad? It means you going back to America without me.'

"But she said: 'It doesn't matter. You've got to do Case Histories again. I want to watch it.'"

Which is how, much to the relief of women everywhere – and doubtless some men – Isaacs came to reprise his role as Jackson Brodie. Not that Emma's motives, it transpires, were entirely selfless. "They watched all six hours: my wife, her mum and nan," says Isaacs. "Emma hadn't seen it, but she had been to the set. She said: 'Put the thing on for Nan, she'll love it.'

"So I put Case Histories on. They watched it and then said: 'Have you got another one?' They started at 8pm and watched until 2am. Emma looked at me and said: 'I think I might be a little bit in love with Jackson Brodie.'"

Not everyone in the family gets to watch, however. "My seven-year-old doesn't watch me shagging," he says. "The therapy bills would be too big."

What about his older daughter, then? Isaacs's eyebrows shoot up playfully. "Does my 10-year-old watch me shagging? No. They do come to the set. It just so happens by accident, not by design, but every time they do I'm either naked or have had my face bashed in."

But Isaacs clearly didn't take leaving his family for three months lightly. On his four days off between filming blocks, he would make the round trip from Edinburgh to LA. "All of my decisions about work are about what is right for my kids," he says. "They are not at an age where I'm going to go off to New Zealand, South Africa, Bulgaria or any of those places for a length of time.

"Other people make different choices and I'm sure it works for them. But, for me, when all of this is gone, and there are no longer clips on YouTube, what there will be is the people around me who either love me and want spend time with me, or think: 'Well, you were never there for me.'

"We were quite old parents, Emma and I. Most of our friends have teenage kids or are empty nesters and I see how much they miss it. This is the golden age. There is no amount of money and no job that can compare to rolling down a grassy hill with your kids.

"When I left them last time, during the gap between filming episode two and three, I was holding my two girls, they were sobbing and my T-shirt was soaked with tears and snot. I thought: 'F*** this.' It's got to be incredibly life changing for all of us for me to be separated from them."

Although that doesn't mean Isaacs has ruled out doing a third series of Case Histories if it's commissioned. "I'd make it work over a summer holiday or something," he says. "We might all be coming back in the summer. My eldest one is at that age where she changes schools, so we could come back [to the UK]."

He is reticent, however, to be drawn on what else is in the pipeline. "The difficulty is when you don't end up doing a job and someone else does. I've had it once before," he says. "A really good friend ended up doing a job I'd passed on. It got to the media – and he's no longer my friend.

"I'm sure that many of the jobs I do other people were in the frame for first, which is why you shouldn't ever speak about it until you are on the set. It's bad karma.

"I said I didn't want to do the job because I wasn't ready for it, I felt I wanted to explore more. It was printed in a red top: 'Why I don't want to be stuck in a dead-end part for the rest of my life,' which wasn't what I said. But to his dying day he will never believe I didn't say that."

While Isaacs mulls over the conundrum of future projects – "somewhere, up on Olympus, the gods are p****** themselves at the way they are moving my chess piece around" – he admits his daughters have taken to California. "They have really flowered as individuals," he says. "They became confident; the biggest and best versions of themselves. The weather helps, too. I was brought up in Liverpool and have spent a lot of time shooting in cold, dark, grey and rainy places."

But when it comes to Hollywood life, Isaacs calls it as it is. "There's some really nice people, there's some w*****s – same as anywhere in the world," he says. "They are just slightly browner and the women's breasts sit slightly higher."

But, when talking about the hellish jet-lag following his whistle-stop hops between Edinburgh and Los Angeles, he lets slip to at least one La La Land quirk: caffeine dependency. "You get back and feel totally out of whack," he says. "I don't even know what planet I'm on. I'm guessing you can't print 'f***ed' in your magazine, but that's the only word to describe it. I'm living on non-taxpaying, Starbucks venti chai lattes. I feel guilty about the tax that, in buying these caffeinated drinks, is going directly to Jersey and the Dutch Antilles, but nonetheless I would be dead without them," he says.

"The [other coffee shops] don't make the rather strange, girly boy drinks I like. I have such a weird, fancy drink, they have to write it down the entire side of the cup and sometimes on their hands as well."

Isaacs admits to being baffled by the phenomenon of celebrities who play out their entire lives on Twitter – "I have mates who tweet their other celeb mates knowing that millions of people are watching that conversation" – but that doesn't mean he's immune from cooking up mischief.

"I was talking to a celeb mate recently, who is married with kids, and we were saying: 'Maybe we should fake a scandal?' What if I fell out of a car somewhere, or we were photographed with a bunch of synchronised swimmers? But actually we just go out with our missuses, have dinner with each other and take our kids to school."

He smiles. "I get to have these adventures: I fight and run and jump and explode things and cry, so I can lead the kind of life that gossip magazines aren't interested in."

The latest series of Case Histories, on which Isaacs is also executive producer, sees comedian Victoria Woods play an accidental kidnapper. "I was completely starstruck," he says. "I'm a huge fan of hers and have been since she was on New Faces."

A host of Scottish names also star including James Cosmo, Gary Lewis, Maurice Roeves and Paul Higgins, who is probably best known for Red Road. "We tried to get as many people with real Scottish accents as possible," says Isaacs. "If there was anything to be criticised last time, there was a couple of people from the Dick Van Dyke school."

Having gone to Central School of Speech and Drama with Higgins he must have some good tales, I cajolingly say. "Yes, but they are all completely unprintable," he says. "My girlfriend shared a flat with his then girlfriend – with very thin walls. I didn't even share that with Paul, so he'll read this and it will explain why it felt like I knew him so intimately."

There is a Facebook page titled Jason Isaacs Would Make A Brilliant James Bond. But the man himself isn't convinced.

"Dan [Craig] and I played lovers for a year in Angels in America so I'm intimately acquainted with James Bond in ways that not many men can say," he deadpans. "I think he's fantastic. He was born to be Bond. I'll be on my Zimmer frame before he gives up on being Bond."

But Isaac admits he wouldn't mind being a Bond villain. What kind of baddie would he be? There's a twinkle in his eye. "That's the exciting thing about my job," he says. "Far smarter people come up with great stories and I get all the credit."

Case Histories begins on BBC One on Sunday May 19 at 8.30pm.