When members of the American security services arrived in London during the early decades of the 20th century, it was not unknown for them to start wearing top hats and adopt English accents.

This infuriated their masters back in Washington (it was all far too deferential) but such aping of British manners reflected an undeniable aspect of the era's intelligence relationship: Britain was seen as the expert and the US was still the apprentice with a great deal to learn.

How times have changed. These days America is clearly the senior partner in a relationship that isn't particularly special any more and a post-imperial island nation off the coast of Europe, while still useful on occasion, can easily be shoved around or flatly ignored.

In his extraordinarily detailed new book, Rhodri Jeffreys-Jones traces the evolution of Anglo-American intelligence relations over the past two centuries.

The basic trajectory is as follows: the US and Britain worked very well together for most of the period – though there were always tensions – but from the 1950s forwards things began to fall apart.

The US grew tired of being regarded as a trainee because this had become a ludicrous perception. It was the world's most powerful nation and it had the most professional and successful intelligence operation on the planet in the CIA.

The student had become the master, and the list of squabbles and mishaps continued to grow: from Suez, to the Cambridge spies, to Britain's noncommittal attitude towards the Vietnam conflict and its overly pragmatic approach to the Communist menace. The net result, on Jeffreys-Jones's account, is that, by the 21st century, the relationship had become "corrosive and obsolescent".

Perhaps those episodes of Spooks in which the US and the UK intelligence mavens don't get along terribly well are not too far from the truth. The old days of "cultural bonding between Oxbridge-educated public schoolboys and Ivy League humanities preppies" are most assuredly behind us.

This is no bad thing, and while the joint triumphs of the past should not be ignored (helping to win the Second World War comes to mind), there was always something lazy and a little too cosy about the relationship: same language, similar culture, comparable values, so what was the point in exploring other security partners with too much gusto?

This had to change and one of the resounding lessons of Jeffreys-Jones's book is that both the UK and the US have to think long and hard about how their intelligence endeavours will evolve in the future. Britain, at least, must tame its Euroscepticism (and it runs deep) when it comes to security initiatives.

One thing will never change, however. If you are a brutal dictatorship then there are few restraints when it comes to spying and gathering intelligence. Anything goes. If you are a democracy, you face a tougher challenge. The perennial conundrum, as Jeffreys-Jones puts it, is "how to make democracy secure without in the process undermining that democracy".

Here we enter perilous territory, especially in these days of transparency, open government and a sensible obsession with civil liberties. Can a Western government secretly subvert the inner-workings of a foreign regime if it believes the ends justify the means? Can it listen in on its citizens' private lives in the interests of the common good? It is very easy, and doubtless rather noble, to take an absolutist position and denounce all such behaviour, but it is also a little pie-in-the-sky.

In order to sustain an efficient security service you sometimes have to bend the ethical rules or, rather, develop an alternative ethical paradigm. This isn't nice but it is necessary and, as this book demonstrates, politics have been engaged in such moral legerdemain since the dawn of human history. It is naive to believe that democracies are capable of acting any differently. It sometimes goes terribly wrong and it can lead to all manner of abuses, but, on balance, it is a useful process. We must, of course, keep an eye on the spooks but we should sometimes rein in our sanctimony.

In Spies We Trust therefore is an excellent title because on spies we depend. They are not all going to be heroic, like the character on page 84 who parachutes into occupied France, gets caught up in a bramble bush and is rescued by a Frenchman wearing, oh yes, a beret. There are, and always have been, clowns, careerists and cads, and the espionage business is not nearly as romantic as its fictive representations suggest. But as that famous figurative line from Twelfth Night puts it, "the rain it raineth every day" and geopolitical downpours are among the worst. We therefore require our figurative umbrellas, – though we should probably try to avoid the actual umbrellas tipped with deadly poison.