SHE was born in Fife to a noble Scottish family, wrote the ballad Auld Robin Gray acclaimed by Sir Walter Scott for which she is still remembered, and lived vividly at the centre of Georgian society. So why is she arguably better known today in South Africa than in her native land? As the biographer of Lady Anne Barnard, it is a question I puzzled over for years.

There is no shortage of signals as to her presence in history. The eldest child of the Fifth Earl of Balcarres, Anne Lindsay grew up in the brilliant milieu of the Scottish Enlightenment, sharpening her wits among the likes of David Hume and Adam Smith. James Boswell used to relate a story from their Scottish tour how this 23-year-old “lady of quality” could hold her own in exchanges with Samuel Johnson.

Escapades there were aplenty. She played a central role in the Prince of Wales’s secret marriage to an alluring widow and went off to observe France during the Revolution while in the throes of a turbulent love affair. All these adventures were set down in the papers, diaries and journals she kept throughout a bewilderingly busy life.

There were also scandals – the reputation the young Lady Anne Lindsay gained in polite Edinburgh society as a heartless coquette and, after her escape to London, an incident involving a dissipated young lord that had her dubbed “the Devil in Scarlet” by Lord Byron’s mother-in-law. These left scars. As became clear when I discovered her papers, it was Anne’s sensitivity about her past that resolved her, in effect, to blot out her story.

But, to go back to the start, how did she end up in Africa?

Anne Lindsay was born in an age when the daughters of impecunious lairds were commonly married off to men of fortune. Such was the fate of her younger sister, Margaret, with whom she shared a primordial bond and who was swept away to London when just 17 by a 40-year-old investment banker named Alexander Fordyce. The marriage was a disaster. Fordyce, the son of an Aberdeen hosier, gambled a mile too far and almost single-handedly brought down the entire banking system in 1771.

From the outset Anne stood her ground against a similar match. When Edinburgh’s literary hostess Alison Cockburn tried to partner her with a wealthy nabob back from India, it set a pattern of resistance. Suitor after suitor was rejected. As Anne explained it, “I am fond of amusement because I am young and have tasted so little that it has not lost its relish yet.”

Her wit and talents – for writing, music and art – were celebrated early on. She sang with the castrato Giusto Tenducci for the Edinburgh Music Society in 1768, and though many of her drawings and paintings remain unseen, some have found their way into the world. It was mainly as a poet, though, that her name endured in Scotland.

Anne was inspired to write Auld Robin Gray while separated from her sister and set this lament for a girl caught in a loveless marriage to a traditional song she and Margaret used to sing together. It became a cultural landmark, being passed down orally until – towards the end of her life – Scott identified her as the author and arranged its publication. William Wordsworth called it one of “the two best ballads perhaps of modern times”.

She was 23 when she met James Boswell and Samuel Johnson at Prestonfield during their tour of the Highlands and islands. Her excitement was still palpable when she wrote to Margaret a few days later. “The figure of the Doctor [Johnson] is a mountain of deformity and disgust. His colour is sallow, his motions paralytic, his sentences pronounced to be repeated. He was silent for the first hour, till he had fed the animal part, which he conducted nastily. That over, he assumed a more questionable shape.”

The fun began when she decided to “rouse the lion”. Banter that Johnson might have been born illegitimate led to her quip, “Would not the Son have excused the Sin, Doctor?” As she reported to Margaret: “The dose took. He became excessively agreeable & entertaining.” Boswell went on to include it in his account of the tour as a jest Johnson used to repeat when in good humour.

But her defiance of convention, her repeated rejection of suitors, sat ill with pillars of respectability such as Lord Kames, Edinburgh’s leading judge and moralist. Kames once sentenced an old chess partner to death with the words: “That’s checkmate to you, Matthew.” When Anne rejected his son, he called her “a witch and a she-devil”.

Attempting to turn a page, she fled to London, but with her credo for love and marriage intact. “Matrimony, I am not ready for thee!” she wrote. “To say Yes to a proposal that would thwart the heart as long as I existed? To cheat an honest man out of the only fortune he can expect to get with me, a free heart? No, I can’t.”

Anne’s family, the Lindsays of Balcarres, could have served as standard bearers for that generation of Scots who carried the Union flag around the globe and, for good and bad, shaped Britain’s empire. She had been told by her father: “You are born after the Union. Scotland is no more and never likely to revive.” Of her eight brothers, four would enter the army, two went to sea and one joined the East India Company. Three died in different corners of the world and a fourth spent years in a Mysore dungeon.

Her own journey to a distant land was slower in coming and due entirely to the partnership she forged with another Scot. Henry Dundas met Anne soon after his election to Parliament. He was near to divorcing his first wife on the grounds of infidelity – though his own liaisons were numerous – and identified this gifted young woman as a partner in his endeavours. Dundas was on his way to forging the Anglo-Scottish alliance that turned him into the “uncrowned king of Scotland” in Pitt the Younger’s government.

Brilliance in company quickly established Anne as a figure in London society. The Prince of Wales became a close friend. Suitors flocked to her door, and for the first time she was genuinely attracted – first to a handsome wastrel, Lord Wentworth, then a damaged, manipulative politician, William Windham. Both love affairs ended painfully but without the disasters that would have come from marrying either.

Meanwhile, however, reputation started to catch up with her. London loved scandals and though Lady Anne Lindsay never gained the notoriety of Frances Villiers or Mary Coke, she had attracted a good deal of gossip while appearing to contemporaries as eccentric – what a later era would have termed bohemian. She lived independently, buying, decorating and renting houses in fashionable locations such as Berkeley Square and speaking what the diarist Lord Glenbervie called “a frank, vulgar sort of half-Scotch”.

All the while she and Dundas remained on close terms, as friends and allies rather than, it would appear, as lovers. He did propose to her at least once before marrying for the second time, another high-born Scottish woman. The likelihood that Anne would find a spouse herself appeared to have passed when, soon afterwards, she received what was her 13th proposal at the age of 42. Andrew Barnard, a handsome but unknown former army officer, was 30. She accepted, and began the biggest adventure of her life.

Barnard had little prospect of gainful employment when Dundas offered him the post as secretary at the Cape of Good Hope. Few aristocratic women of the time opted to sail across the world in dangerous and insupportable conditions in order to live in backward hellholes, and the assumption was that Anne would remain at home. When she told Dundas she would be accompanying her husband he snorted: “Good God! You go to live with the Hottentots?”

In the event, the Cape brought Anne the happiest years of her life. She and Andrew lived in a simple cottage called Paradise at the foot of Table Mountain where she kept a small menagerie of wildlife. At the same time she acted as the governor’s hostess, extending hospitality to suspicious Dutch residents and nabobs passing to India alike. Her egalitarian nature found expression in dealings with the Khoikhoi population and slaves. The most astonishing poem ever to come from her pen dates from this time and concerns the love between a white woman and a slave.

She had a wider vision too. In 1799, when the Barnards set off on a wagon tour of the interior, Pitt’s government saw the Cape as no more than a strategic bastion to protect its vital shipping network to India from the French. Anne did her best to convince Dundas that Africa too had potential. “Here is scarcity, but here will be plenty,” she wrote to him. “It is in the power of activity to make this the finest scene in the world by planting.”

Over the five years she spent at the Cape, Anne wrote constantly to inform Dundas of what she called “every point which regards publick concern”. She was as good as her word, and ultimately it was in large part thanks to these visionary letters that she can now be restored to history.

Andrew Barnard died at the Cape. It is clear from their correspondence that the marriage had been supremely happy and nothing could replace him in her life. Anne lived at Berkeley Square until her death aged 74 in 1825, devoting her final years to the creation of six volumes of memoirs in which she recorded her life with wit and irony, along with a searing honesty that almost places her in another age from that other indefatigable memoirist, Fanny Burney.

One obstacle has until now stood in the way of wider recognition. Anne issued an edict before her death that her writings, and specifically the memoirs, were for the eyes of family only. “As far as a departed person can debar anything,” she wrote, “I utterly debar the publication now or ever of any work of mine.”

That changed when the South African Library acquired Anne’s letters to Dundas, and made them available for publication in 1973. This created a tremendous stir among historians and readers with a general interest in the South African past. Academics from South African universities started to beat a path to Balcarres, the main repository of Anne’s papers, leading to the publication of her Cape diaries and journals.

It was as a reader familiar with the South African writings that I approached the Earl of Crawford and Balcarres four years ago, requesting permission for access to her papers. I was born at the Cape myself and had in mind a book based on her life in Africa.

As it transpired, the memoirs – still unpublished and virtually unexplored – opened the door to exhilarating and even more astonishing places. Anne kept my interest all the while, and had surprises in store, right to the end.

Defiance: The Life and Choices of Lady Anne Barnard by Stephen Taylor is published by Faber, priced £20.