THE metal door clangs shut, plunging the tiny, miserable space I have just entered into inky darkness. I am imprisoned in a cell buried below an old “boozing-ken” close to the Old Bailey – “the great theatre of crime and punishment” – which stands on the site of Newgate prison.

Thankfully, I am not alone in the murky cellars of the Victoria Tavern, despite a mischievous barman’s attempts to incarcerate us. I am accompanied by the novelist Jake Arnott, who is taking me on a three-hour long, walking tour of Georgian London’s terrifying underworld, from gaol to gin palace, from coffee house to bawdy house, from the site of stinking sewers to the stately Surgeon’s Hall, where the cadavers of condemned villains were “anatomised”.

We visit churches, turnstiles and the goddess of the gutter as we follow the underground bed of the filthy River Fleet. We make many detours down labyrinthine alleyways. I am introduced to a “Thief-taker General”, described by Arnott as “a cross between Deacon Brodie and Donald Trump”. And we wander enthusiastically around the Hundreds of Drury – the labyrinth of infamy around Drury Lane once known for vice and prostitution – to our grim journey’s end on a traffic island at clangorous Marble Arch.

Here, a sad little plaque, surrounded by three sickly oak trees, marks “The Site of Tyburn Tree”, the gallows otherwise known as “The Fatal Tree”, the title of Arnott’s latest novel.

Arnott, 55, who was born and educated in Aylesbury, originally pitched the book as “Moll Flanders meets A Clockwork Orange”. Set in the Beggar’s Opera world of the 1720s shortly after the South Sea Bubble has burst, it’s a dazzling mix of fact and fiction told in thieves’ cant or “flash”, a gloriously inventive street slang to which Arnott supplies a six-page glossary, although the reader rapidly attunes to the lexicon – bene (good), phiz-monger (portrait artist), queer-bluffer (devious inn-keeper) ... The latter could, of course, apply to the fellow who attempts to bang up Arnott and me in those doleful cells.

The Fatal Tree is the Hogarthian tale of a Harlot’s Progress. Elizabeth Lyon is a comely, respectable country lass seduced by her master’s son. She fetches up in “Romevillle” – underworld London – where she becomes a prostitute at Mother Breedlove’s Vaulting School. She meets the sinister thief-taker, Jonathan Wild, who names her Edgworth Bess, “that lewd soul”. Soon she’s ducking and diving, boozing and whoring while thieving with the likes of Jack Sheppard, with whom she falls in love. They marry but their romance is inevitably doomed.

Bess, Sheppard and Wild are, of course, real figures. Sheppard, a true Jack the Lad, was a Houdini-like petty criminal who escaped Newgate prison twice. “He’s the inspiration for the character of Macheath, the captain of a gang of robbers, in John Gay’s 1728 The Beggar’s Opera, which in turn spawned Brecht’s The Threepenny Opera,” notes Arnott.

He discovered Bess while researching the 18th-century culture of celebrity criminals for whom he has a penchant anyway. His acclaimed debut novel The Long Firm (1999) was set in the East End gangland among the likes of Jack “The Hat” McVitie and slum landlord Peter Rachman in the 1960s. Arnott had been asked to participate in an exhibition, Georgians Revealed, for the British Library and he found Bess, alongside Sheppard and Wild, in the Proceedings of the Old Bailey, 1674-1913.

There are few mentions of her in these court reports but then Arnott read that Sheppard had said of her: “A more wicked, deceitful & lascivious wretch is not known in England.” Arnott was captivated. “I thought, here’s a femme fatale, a bad girl. But she turned out to be much more interesting than that. She’s not just Jack’s sidekick. She has such agency, because she’s instrumental in helping him escape. I wanted to know the story behind all that. They are Bonnie and Clyde and I’ve always been interested in the underworld anyway, because it’s a dark reflection of the over-world, which interests me too.”

While Daniel Defoe is the alleged author of Sheppard’s autobiography, which was sold at his execution in 1724, attended by some 200,000 onlookers, Arnott has given Bess her own amanuensis, Billy Archer, a hack writer grubbing a living in Grub Street, where he hangs out with the great wits of the day: Swift, Pope and John Gay.

The fictional Archer’s own tragic story is woven into Bess’s narrative. He’s a homosexual or a “molly”, a frequenter of molly-houses, who also has an unhappy love affair, and a denizen of Wild’s den of thieves.

“Criminal narratives were all the fashion providing much work for hackney-scribblers like Archer and there was an explosion of new media,” explains Arnott. “Coffee houses were known as penny universities. For the price of a cup you could get all the journals of the day – which is like internet access because we also get that by buying a coffee. There were pamphlet paper wars providing the buzz of gossip. A pamphlet could be published in a day. Someone would be attacked in print; within two days they’d answer back. It’s like internet trolling. It seems too much of a clever thing to say but the pamphlets were also small, about the size of a tablet.

“Look at Hogarth, he invented the graphic novel. In my book the observant reader will spot a number of allusions to his Harlot’s Progress.”

Which brings us to Jake’s Progress as he leads me to a “gospel-shop”, the Church of St Sepulchre-Without-Newgate, where there’s a gruesome secret: hidden behind a pillar is the handbell rung 12 times at midnight on the eve of an execution, “whose sound carried through an underground passage to the very cell of the doomed wretch”.

“Padding” – walking – to Holborn Viaduct, we pass Ye Old Mitre, near the site of Mother Clapp’s molly-house. Across Hatton Garden we head onto High Holborn, where Sheppard planned to make his last escape at Little Turnstile, an alleyway off the main thoroughfare. He was to have made his way to Lincoln’s Inn Fields. Once located next to the Old Bailey, Surgeon’s Hall is now the Royal College of Surgeons with its Hunterian Museum.

“There’s someone I want you to meet here,” says Arnott, bounding up the sweeping staircase. He’s tall so he blocks the glass case, then he stands aside and with a theatrical gesture – Arnott has had many jobs including being a mortuary technician and an agitprop-actor and even played a Mummy in The Mummy – he exclaims: “Meet Jonathan Wild!”

I am face to face with the skeleton of the thief-taker and gangland boss of a nest of villains, who wove a spidery web of theft and venery across London from his office at the Old Bailey. A man of dubious morals – “the nearest you have to him in Scotland is Deacon Brodie. He’s a bit like Trump, promising that he would drain the swamp”.

Wild was hanged, however. “His remains were dug up and his cadaver turned over to the surgeons,” says Arnott. “The great fear among criminals was that surgeons would take their bodies and dissect them – this is a century before Burke and Hare. Look, next to Wild’s skeleton is a print of Hogarth’s The Reward Of Cruelty, showing a surgeon watching the viscera of a murderous highwayman being removed.”

Time for a restorative drink, so we repair to the Angel Tavern, next to the site of the old Bowl Inn, in the slums of St Giles, the last church before Tyburn. Here, the condemned were offered their final drink, a tradition started in the 15th century.

Leaping onto a 21st-century cart (a Central Line train from Tottenham Court Road to Marble Arch) we avoid “that dismal throughfare that is Oxford Street”. In the dreich, late-afternoon smir we stand where the gallows cast their fearful shadow and where Bess’s story begins and perhaps ends. Notes Arnott: “The procession to Tyburn takes a central place in my novel. It is the fate of so many of its characters after all, and it formed a fearsome spectacle for the city of that time. In this dangerous world all dwelt in the shadow of the fatal tree.”

The Fatal Tree, by Jake Arnott, is published by Sceptre, £16.99.