Signposts near the Border south of Coldstream point to the site, their crossed swords the tourist equivalent of a skull and crossbones.

The road narrows to a tight country lane, leading to a car park where visitors are warned to lock their doors in case of theft. It is the only hint of menace in an otherwise idyllic setting. Indeed, such is the blood-soaked reputation of the Battle of Flodden that the tranquility and loveliness of this corner of Northumberland are almost startling.

Rising against a lush backdrop of rolling fields and trees, where nesting birds dart between hedgerows, two gentle green hills face each other. A narrow burn divides them. Burbling along between crops and grasses, home to minnows and dippers and dragonflies, it is a pleasant spot for a picnic. But on the afternoon of the most devastating military defeat ever suffered by Scotland, this stream was to become as deadly as the killing fields of Passchendaele. Racing with storm water, a treacherous quagmire spreading from its banks, it caught the advancing Scottish troops unaware. As the first rows sank into the mud, those charging from behind fell on top of them. This was the beginning of the end. Scarcely two hours after the battle began, most of the Scottish army and their king were dead, and with them the future of Scotland.

For some Scots, the memory is still raw, a tragedy and an insult to national dignity that remains painful almost 500 years on. For most English, however, it merits little more than a footnote in the annals of history. Flodden is just one of countless battles from which they emerged triumphant. It changed nothing. Whereas for Scotland, it changed everything.

On September 9, 1513, when James IV stood at the head of his great army and looked out at the exhausted English ranks, the outcome must have seemed fairly sure. What had probably begun as a diversionary exercise, to distract the young Henry VIII, who was already fighting in France, had swiftly developed a momentum of its own.

For years, James had been pursuing a dangerous but deft diplomatic policy of keeping his friends close and his enemy even closer. His friend was Louis XII, whom he promised to help as best he could in his campaigns against the English. Meanwhile, as the brother-in-law and neighbour of Henry VIII, he had signed a treaty of Perpetual Peace with England. Any backsliding on that deal would be potentially disastrous.

Quite why James decided to pit himself directly against the English is not wholly clear, though his affinity with the French, and his growing belief in his own claim to the English throne, surely played a part. What is certain, however, is that had James kept calm in the days and hours before battle, his superior numbers and peerless defensive position should have guaranteed victory.

Instead, for reasons that perplex historians to this day, shortly before battle commenced he moved his army from the impregnable Flodden Hill and set up on Branxton Hill, a couple of miles away. This entailed dragging enormous new cannons into place so late in the day that, with their range misjudged in the panic, they proved useless, wildly overshooting the enemy.

In addition to this, the terrifying 18-foot pikes James's soldiers carried were quickly found, in close combat, to be suicidally lethal, no match for the short but deadly English billhook.

Battle began in the early afternoon of a Northumbrian day so dreich it might have been winter. By dusk, it is thought that more than 10,000 Scottish soldiers had been slaughtered, James IV and his son Alexander cut down with the rest of them. Along with thousands of foot soldiers from all across the country, Scotland's aristocracy, its leaders, senior clergy and counsellors were almost wiped out. With the statesman class annihilated, so was any chance of Scotland regaining its role as an outward-looking and sophisticated country. As the most famous Border lament later mourned, "The flowers of the forest are a' wede away." With their loss, Scotland returned to the dark ages.

It could be argued that, had the Battle of Flodden been won by the Scots, the landscape of British political history would be unrecognisable today. With the help of Louis XII, James might have staked a claim to the English throne, and had he prevailed, the axis of power might have lain in the north, not the south. Had that happened, Scotland might still be an independent country, with no need of a ballot box to return us to the days before the Union in 1707.

That scenario, however, depends on a cataract of ifs and maybes. Too many, probably. What intrigued me about Flodden was not the constitutional question, but the psychological. What did the survivors feel about what they had seen and done? How did those who lost their loved ones in the battle cope, or carry on? And what kind of country had Scotland now become?

If you were brought up near the Border, as I was in Dunbar, and if, like me, you had a father who breathed history, it was impossible not to learn something about the area's past. For some reason, however, the story of Flodden made a particularly strong impression. Perhaps it was the sight of tattered battle flags hanging in St Giles's Cathedral. When I first saw them, they swayed in the draught as if held in a palsied hand. I liked to think I could see spatters of blood from the men who'd been carrying them.

I am not sure, though, what turned the bare bones of this story into the idea for a novel, so many years later. It may not be a coincidence that I began to think about Flodden, and those responsible for this catastrophic mistake, as the war in Iraq grew bloodier. Although we live in a so-called democracy, world leaders such as George Bush and Tony Blair are the modern equivalent of Henry VIII and James IV, able to wage war regardless of public opinion. As the political firestorm over Iraq raged, it became clear that the closest advisors of presidents and prime ministers, be they Donald Rumsfeld, Alastair Campbell or their wives, play a part often invisible to outsiders yet crucial in the outcome of events.

I began to wonder. After centuries of being reviled for taking his country into an unnecessary and crushing battle, James IV's reputation has in recent years been restored. He is now widely regarded by historians as a monarch who put barely a step wrong in his short reign, who at the time of his death was transforming a backward northern country into a renaissance power worth reckoning with. If this was true, then some of the answer for Flodden must lie with his inner circle. Very quickly this led me to his secretary.

Patrick Paniter is a shadowy figure in Scottish history, yet he played a significant role in James IV's court. An intimidatingly astute diplomat, he conducted most of the king's political correspondence. Wholly trusted – he had also been tutor to James's eldest son – he was often left to deal with matters as he saw fit. Despite his scholarly mind, it was Paniter who was in charge of the cannons at Flodden. Moreover, of the high-ranking, well-educated men who stood on that hill, he was one of the few to escape unbloodied.

The novel After Flodden came to life one winter afternoon in Edinburgh's Central Library, as I read the letters of James IV's court. Among them I found Paniter's occasionally high-handed directives to kings and courtiers across Europe, to James's subjects, and to Alexander, James's illegitimate first-born, and Paniter's protege, for whom he had a great affection, even though he scolds him for neglecting to write back to him.

As the idea of Paniter's political influence grew, so did my interest in the Border country around Flodden, and beyond. James IV and Henry VIII often complained that this was rebel territory, the Borderers ungovernable and answerable to none, least of all a king. The story of these fiercely independent people, and their part in the unfurling of Scotland's violent affairs seemed to me an essential part of any account of Flodden, and what followed.

So too was that of the wives, sisters and daughters whose lives were damaged or destroyed by the battle. In Louise Brenier, a woman who does not know if her brother died on the field or was taken prisoner, I found a character whose courage and determination stand for so many of the women of her era. Commoner, high-born or pauper, they were exceptionally brave and resourceful.

Scotland's history can be read like a book whose spine is broken. Before Flodden things were far from perfect, but there was hope and confidence, and an unquenchable spirit. After Flodden, Scotland was never to be the same again. We can speculate endlessly on what the defeat has cost us down the centuries. Instead, I have tried to picture its impact on the people and players of that time. n

After Flodden by Rosemary Goring is published by Polygon on June 6, priced £14.99. Herald readers can purchase it for the special price of £12.50 (including P&P in the UK) by calling 0845 370 0067 (office hours) and quoting ref HF613.