“There are spies all around us.” It's one of those whiff-of-paranoia comments that conspiracy theorists wheel out in the pub or wannabe hacktivists spray around in the Dark Web.

That doesn't make it any less true, of course. Just last week, China announced a “spy hotline” for people to call if they suspect foreign nationals of conducting espionage activities – activities such as bringing up controversial topics at parties and then not joining in the conversation. Or having lots of money but only a very vague job title. Suspicious? Us?

Track back a little from the headlines to the television set and the spies are there too. Next up on our platter of fictional treats is London Spy, a five-part television drama for BBC Two. It stars Ben Whishaw – currently playing Q in Spectre, of course – as a hedonistic young gay man who becomes embroiled in a spy conspiracy when his closeted lover, an MI6 officer, is found dead in circumstances that are sordid and scandalous. Or appear to be, anyway: one of writer and series creator Tom Rob Smith's starting points was an old CIA manual he came across some years ago. It shows how to make a death look accidental by weaving it into the narrative of the victim's life.

Jim Broadbent and Charlotte Rampling also star, the latter playing what Smith calls “an enigmatic, formidable figure who seems to be orchestrating the conspiracy”. Think of Judi Dench as M then imagine her nastier, taller, steelier evil twin.

Smith, who is gay himself, is the author of four novels including Child 44, released as a film earlier this year with Tom Hardy playing hero Leo Demidov, a Soviet-era policeman tracking a serial killer. His most recent novel, The Farm, is a psychological thriller, and he started his career working on Channel 5 soap Family Affairs.

It's not an obvious CV for the writer of a series like London Spy but spy stories are in fact a sort of literary lodestone for Smith and behind his love of the genre are some very personal reasons.

“I haven't had the John le Carré life, but I have had a life of wondering how I am a spy on some level – and that connects with being gay and leading a double life in the sense that you have this persona,” he explains. “At school I was terrified. I couldn't even comprehend it [being gay] so my way of dealing with it was to say, 'I'll be a much more convincing liar if I can convince myself that it isn't true.' So on one level I shut that whole part of my brain down and I became a spy to myself. I was living a cover that I started to try to believe in order to make it more convincing. That's my reason for getting into spy stories.”

But what about the rest of us? How does he account for the wider public's never-ending appetite for spy stories? One reason, he says, is the way they lend themselves to great action sequences and moments of heightened drama.

“The second reason, and the reason they're so popular and persistent, is because they do resonate with the everyday. It's not just a question of thinking, 'I'm not a spy, I'm only watching this because it's an interesting story'. There is always some sense of, 'Could I be that person? Could I pretend to be someone else to get some information? Would I be a good spy?' And I think when people watch spy stories they're always asking that question and it must connect to their life. And even if it's only fleeting, it's always there on some level … I think identity is at the centre of spy stories.”

Accordingly, Smith says London Spy isn't a drama about the intelligence services nor is it an attempt to explore their everyday workings or capture their reality. Instead “it's about how spy stories are relevant to everyone regardless of whether you're connected to the intelligence services or not. I'm interested in the spy genre in the domestic world. I'm interested in how it connects to our everyday lives in terms of our families, our lovers. Do we know the person sitting opposite us in the workplace? Do we know who's plotting against us? Are we pretending to be someone to one person and a different person to someone else?”

His answer – certainly the one he lays out in London Spy – is yes, probably.

Elsewhere, there's plenty more evidence of the extent to which we remain fascinated by the shadowy world of espionage.

Exhibit A: Homeland. Now into its fifth season, the American TV drama is no less gripping in its detailing of CIA operations in Berlin and Beirut than it was when covering the covert missions in Afghanistan, Pakistan and Iran ordered by the morally ambiguous Washington spy masters in previous series. This time it's all about regime change in Syria.

Exhibit B: James Bond. You might have noticed that he's back behind the wheel of his Aston Martin DB10, this time for a 100-mile-an-hour chase through the streets of Vatican City in Spectre, film number 24 in an apparently never-ending series of high octane spy thrillers. Keeping things bang up to date, the sub-plot turns on the tension between Bond's old-style intelligence gathering and the digitised snooping preferred by a new broom in the spook cupboard.

Exhibit C: Cornwell, David, better known by his literary pseudonym John le Carré.

Still rated one of the greats over half a century after he published his game-changing novel The Spy Who Came In From The Cold, his stories continue to be read, filmed and relevant. Communism may now be just a bad taste in the mouths of Eastern Europeans, but le Carré has continued to tackle modern ills: Big Pharma, for instance, or state torture and extraordinary rendition. No surprise, then, that a biography of him published last month is already a best-seller.

Adam Sisman is its author and, unsurprisingly, he agrees that this question of identity is a core part of the appeal of the spy genre and thinks it's particularly relevant to both le Carré's fiction and the man himself.

“There's a wonderful quote in Absolute Friends where le Carré encapsulates that in a sentence,” he says. “There's certainly a sense of theatre and a sense of acting. That's dramatised particularly in The Little Drummer Girl where an actress is called upon to play the part of a Palestinian sympathiser and get close to a Palestinian terrorist so Israeli intelligence can winkle him out. They call it 'the theatre of the real'. As for le Carré himself, well he is a man of many masks and the metaphor that's used in the credits at the beginning of Tinker, Tailor, Soldier, Spy – of the Russian dolls – can be applied to le Carré himself.”

The spy story, he adds, “is a dramatic hidden world where individuals can apparently play a part in a way that they can't in more obvious conflicts, where a superpower conflict during the Cold War for example can be fought out in the assassination of an individual on Hampstead Heath.”

But is it just us Brits who love a good spy story? Given that the giants of the genre, Fleming and le Carré, are both British, it seems likely. You could throw in other candidates for the pantheon but again they're either British or they made their name writing in English: Joseph Conrad, John Buchan, Eric Ambler and Graham Greene would all feature.

Granted, two of the best big screen spy thrillers of recent years were directed by a Dutchman and a Swede, but those films by Anton Corbijn and Tomas Alfredson had one common denominator: they were both based on le Carré novels.

Corbijn cast Gary Oldman as George Smiley in his stylish remake of Tinker, Tailor, Soldier, Spy while Corbijn gave Philip Seymour Hoffman his last screen outing in his adaptation of the Hamburg-set A Most Wanted Man. Conversely, it was a British director, Paul Greengrass, who added punch and weight to the film adaptations of American author Robert Ludlum's Bourne novels.

For film critic David Denby, writing about John le Carré in The New Yorker last year, the reason his fellow Americans don't have the same affinity with the spy genre is because “they fail to enjoy spying as a treacherous game”. Instead, “they think they are saving the world, whereas the Brits know that, apart from Britain’s dwindling interests, there’s nothing to be saved, just the endless struggle itself, well or poorly joined”.

Anyone who has read John le Carré knows that the author’s scepticism about the American establishment is even more of a constant than his dislike of communism. In part it’s down to American methods, but also to gung-ho nationalism and the implied certainty that the American way is best.

Le Carré, on the other hand, “shows the moral murkiness behind the ideological slogans”, says Sisman. “And in the character of George Smiley, in particular, he has someone who's always worrying whether the sacrifice involved, in terms of the suffering of innocent individuals, is worth the cause. That dilemma – between the cause and the individual who gets crushed – is there in a lot of his books … So I think le Carré's novels are a protest against ideology and fanaticism at the expense of human individuals.”

So as well as issues of identity and role-playing in personal relationships, that idea of moral ambivalence is another appealing factor of the spy story.

Of course there are those who enjoy gung-ho nationalism, who venerate loyalty and detest moral ambivalence and treachery. These people prefer black and white to endless shades of grey and for them there is Ian Fleming and James Bond.

You won't be surprised to learn that le Carré isn't a fan. “He's very dismissive of Ian Fleming,” says Sisman. “He dislikes the macho posturing, the unquestioning support for the cause of the West." But Tom Rob Smith is much less hard on 007 and on his creator. “I love the whole range [of spy stories], as contradictory as that might seem,” he says.

Smith wrote the introduction to the Vintage edition of Ian Fleming's From Russia With Love and has always liked Bond in particular.

“Bond is a real lover of life. Fleming is always going on about how much he loves his breakfasts, for example. It has become a cliché now, that Bond has great taste in things and perhaps we're overly familiar with it. But that does draw from an interest in energy: yes, he's a killer but he loves the world. He loves food, he loves drink, he loves women, he loves cars. That's why he's protecting it, as opposed to the people who despise the world on some level. I love that energy. It's very powerful, so I've always loved Bond.”

Ultimately, then, perhaps we enjoy stories about spies and spying because we're the best at telling them – and perhaps we're the best at telling them because it's the genre most closely suited to explaining our slightly schizophrenic national psyche to ourselves and to others. We can love James Bond and George Smiley because even as we stand behind the flag we can mock it and question it: because even as we settle down with a loved one to watch a television programme – London Spy perhaps? – we can dissemble and obfuscate and lie.

London Spy starts tomorrow on BBC Two (9pm). John Le Carré: The Biography by Adam Sisman is out now (Bloomsbury, £25)