The best book ever written about spies and spying, in the sense that it is the most revealing about the murky psychology and the weird transvaluations of espionage, is a novel, John Le Carre's The Tailor Of Panama.

Max Hastings runs the master very close indeed. The Secret War has its own astonishing cast of characters and its remarkable human narratives, some as familiar and overplayed now as Alan Turing and Bletchley Park, others only recently mined out of foreign archives or rescued, like Turing's Bletchley colleague Bill Tutte, from an undeserved obscurity. It's astonishingly detailed – where does he find the time to research so many big books? – and one of the first to apportion space evenly to the wildly different intelligence networks of the Allied and Axis powers.

There's a strong temptation simply to rehearse some of Hastings's best stories or give thumbnails of some of his more exotic protagonists, and leave an impression of English Patient plots, crossword puzzlers and chess masters, Mata Haris, lonely alcoholics with no discernible moral fibre let alone patriotism, genius and sheer raw luck in magnetic conjunction, and fox-hunting boffins straight out of the pages of Evelyn Waugh. And there's some temptation to suggest that Hastings delivers a now familiarly subversive view of the intelligence communities which is consistently premised on the idea that the wrong people were invariably recruited for the most important jobs.

His book is more important than that, though, and gives the most realistic and nuanced account we've yet had in accessible form of the limitations of espionage. Early on, Hastings suggests that "only maybe one-thousandth of one per cent" of wartime intelligence contributes in any way to battle realities. That's a facer, given the effort some countries expended to gather it, though that effort was dramatically altered by a new and overwhelming military dependence on radio communication – 'sigint' somewhat displacing old-fashioned 'humint' and paper – which could simply be butterfly-netted out of the air and handed to the decoders. Somewhat later on, though, he gives the hard word on what really came out of our own most cherished wartime espionage effort. "What was done at Bletchley was, indeed, miraculous, but the code breakers were never able to walk on all of the water, all of the time." Indeed, and the result of that, despite Bletchley's very material contribution to the Battle of the Atlantic, is that a lot of good men had to drink seawater, laced with oil.

The odd thing about the now settled Bletchley narrative is that it misses a lot of fascinating detail. We've been given to understand that Enigma was the code of codes. It turns out to have been considerably less complex than the Lorenz Schlusselzusatz; less catchy for the thriller writers, maybe. And when it comes to sheer cerebral effort and brilliance, Bill Tutte's deduction that the noises he was hearing – and this sounds like the old thriller chestnut of the blindfolded hero counting bumps in the road – could only have come from a teleprinter with two sets of five wheels, one of which 'stepped' irregularly, with 501 settable pins and two further motor wheels. Sounds easy when put like that. Some developments came from milder and perhaps more human perceptions: like the 21-year-old Cambridge mathematician John Herivel, half-asleep one night, suddenly thinking that an Enigma operator as tired as himself might just cut corners and use yesterday's settings again. 'Herivel tips' kept Bletchley going from the fall of France to the Battle of Britain.

One by one, Hastings knocks down rooted misconceptions. It's a reality of war, for instance, that most signals intercepts and decodes are only of retrospective, not real-time significance. In the same way, supposedly 'visionary' predictions of future events were almost always highly speculative or tentative at the time and only invested with predictive magic because they happened to be right. Churchill seems to be the exception. He was almost invariably right, especially when he stood alone. On the other hand, some glaringly detailed intelligence, like the German invasion of Russia – and Hastings rightly and repeatedly insists that Barbarossa was the single key moment of the war – was simply ignored. The psychology here is fascinating. Stalin preferred to believe British intelligence leaks sent by Whitehall traitors to the product of his own remarkable intelligence network. He believed Hitler, at that point still an ally, would attempt to knock out Britain first. A simple telephone call to East Prussia would have confirmed a different reality.

As illustration of just how differently the combatants approached intelligence, the Japanese seem to have put in extraordinary effort before their sneak – but also much predicted – attack on the USA, but then largely abandoned intelligence-gathering during combat. It may have hampered them; it may even have helped the Bushido steamroller gain momentum.

Hastings overturns long-standing myths. The idea that Admiral Canaris was at the head of a noble, old-German resistance to Hitler is dismissed as sentimental nonsense. The treachery of a few Cambridge spies is set against the literally hundreds of Americans who allowed Popular Front sentiment or ethnic loyalty to overcome patriotism. This is as close as anyone can decently come to justifying McCarthyism but, yes, there were would-be reds under lots of beds.

The underlying message of The Secret War comes in a series of overlapping perceptions, at least some of which can also be found in The Tailor Of Panama: the idea that fantasy is the essence of intelligence; the notion that secrecy is always more important than accuracy; the recognition that a “failure of intelligence” is usually a failure of will, this particularly so where the “Nazi threat” is concerned.

The remarkable Donald McLachlan, author of Room 39 about naval intelligence, said that "career officers and politicians have a strong interest in cooking raw intelligence to make their masters' favourite dishes". Intelligence 'gathering' is a misnomer; it's almost always an artefact. In the same way, as Hastings wisely puts it there is in the corridors and sub-basements of power "an institutional precept that no intelligence assessment could be countenanced by policy-makers which ran contrary to a desired national course". Governments sex-up the data one minute, and put telescopes to blind eyes the next. These aren't abstract lessons; they apply to Iraq and Daesh, to Ukraine and the Baltic; they come bang up to date and home to roost.

Secret War: Spies, Codes And Guerrillas, 1939-1945, by Max Hastings is published by William Collins, priced £30