Andrea Stuart's family history, Sugar In The Blood, is a superb feat of research and memory, so I know it can't be her fault that she's late for our interview.

Quarter of an hour after we are due to meet in – as I think – the British Library, she calls to let me know she's in the British Museum. Fortunately for me, Stuart is not into blame. In fact, when we finally meet, she is the most laidback of writers, an effervescent and youthful-looking woman of 50 who, despite having two young children, is as serene as if she has just stepped out of a spa.

Stuart was born in Jamaica but was 14 when her doctor father moved his family to London. Educated at a private girls' school, she studied English at the University of East Anglia, French at the Sorbonne, and is now writer-in-residence at Kingston University, London.

Sugar In The Blood, her third and most ambitious work of non-fiction, is the sometimes harrowing, always fascinating story of Stuart's family, from its first recorded member to the present day. What makes it unusual is that Stuart is descended both from slave owners and from slaves. Thus this deeply disturbing period of history is described feelingly from each perspective.

Asked how she felt embarking on such a painful and potentially controversial work, she replies without hesitation: "I kept thinking this book tells a story that Britain does not want to remember." Yet although she knew the history of Caribbean slavery, she was not prepared for how nasty the facts would be. "I can't lie," she says, "I don't think I could do this book again. There were moments which were really upsetting."

Her story opens with George Ashby, an ancestor on her mother's side, who in the late 1630s sailed from England to Barbados. By his death, Ashby had established himself as a small landowner. Future generations added to his sugar cane acres, so that by the early 19th century the most swashbuckling of the family, Robert Cooper Ashby – who married an heiress – had become a very wealthy planter with an army of slaves. A staunch anti-abolitionist and a man with few scruples, Cooper Ashby expected his genteel wife to avert her eyes from his many slave mistresses and their illegitimate offspring.

Stuart's father's side of the family is descended from one of Cooper Ashby's slaves. This part of her background was far harder to research, she says, partly because there were few records of slaves, and partly because, as she recalls, "my father's family, I suspect, had a darker remembrance of slavery. Their side of the family was like, 'Oh no, let's not bring up the past, let's move on.'"

Who can blame them? Among the misery Stuart uncovered was the discovery that the slaves' ordeal did not begin with the grim journey across the Atlantic. "It was the bit before, and the fact that many of these slaves could have been imprisoned for two years before they even saw a ship. There was the terrible number of deaths on the way, but also that grinding process of dehumanising and debasement and illness.

"The middle passage seemed to me to be suffering enough. I was prepared for that. What I hadn't been prepared for was the many, many months before. There were bits of it, like research on slave punishments, I remember just bursting into tears, thinking this is so horrible."

Fortunately for the reader, Stuart offers enough information, but never makes the book unbearable. In fact, it's astonishingly readable, her elegant, thoughtful style the perfect foil for her often shocking material. As one might expect, she is profoundly angry about the ongoing impact of slavery, which continues to blight the black community, in the Caribbean, America and Britain. This legacy, she says, includes "the whole idea of the invention of racism, the creation of disadvantage amongst black British subjects".

That harm, and the attitudes that caused it, she adds, are still evident in the way Britain is run today. For instance, it was easier for politicians and activists such as Wilberforce to abolish the slave trade than to abolish slave ownership. "It's one thing to get rid of a few vulgar roughnecks who you would never have in your house under any circumstances, but to have somebody who's gone to school with you at Eton or Westminster, and university with you, a dear friend from childhood, who read the same books and lived on the same street as you -"

She gives a grim laugh. "We haven't moved on very much from that, have we? Our government, for example. We have a new aristocracy, but pretty much the same idea, with Cameron and Osborne, a whole caste of people who all were at Eton and are very much part of the same story. If you look at 18th-century England, and the preferences, the cronyism – we're no different now."

Stuart sounds disheartened. "I battle all the time with this issue. Somebody was talking to me about affirmative action, and said, 'People say you can't have that in Britain.' The truth is, we already have affirmative action very firmly for a particular social group in Britain. We say this with such piousness, 'Oh, affirmative action would be a terrible thing. We want to be fair.' And I think, when has British politics been fair? This country has such a history of these kinds of nepotism and cronyism, and we really, really have to find ways around it."

And yet, for Stuart herself, the consequences of uncovering her family's role in the sugar trade have been positive. "Understanding the backstory of my family and my whole community has been incredibly liberating and useful - I realised really rather proudly how incredibly we as a community have contributed to the wealth and the power of Britain, and how much we have earned the right to be here, and to be part of the story of British life.

"I think I've always felt quite ambivalent about taking part in British life, because I always felt I wasn't really British in some way. Also, I came here when I was 14, so my attachment to the Caribbean is quite visceral, and it's where I grew up. I think it's taken me a long time to realise we've earned our place here with blood and tears, we've generated massive wealth, we've created Britain partly as Britain is. It somehow made me feel stronger in my Britishness, my right to be British."

The book had another dramatic effect on her, as she describes. "I came out of it being incredibly proud to be a descendant of a slave. The sheer visceral ability to cling on to life, to hold on to life, and then to carry on living it, to learn to love and find joy, holding on, through a slave system that went on for 300 years, seemed to me such a remarkable feat. I look at my grandfather, only a step down from slavery, he was just one generation away. Really, the shadow is very short that way. My family are very disciplined and hardworking. I thought, what a miracle this all is."

Does she ever wonder how she would have coped as a slave? "I know I would have been one of the ones who'd have died really fast," she replies. "I would not have been a survivor. I'm kind of weedy. I get ill all the time. Physically, I can see it, in my father, that incredible tenacity and strength, but I would have died within months of capture."

There was nothing remotely weedy, though, about writing this book. Nor in her own life. With a seven-year-old white daughter by her partner, Tara, they then adopted a black girl, who is now three. It seems natural to ask whether with this family Stuart was trying to challenge and heal the old black and white divisions. She pauses, taken aback by the idea. "You know, I'm a little floored, because I hadn't thought of it like that, but it makes perfect sense, psychologically -

"I always thought I would probably have an unconventional family. It didn't surprise me that it was a little bit different. One of the things I thought about the book is how people would talk about slavery as a black story, when clearly it's not a black story – by definition it's a white and black story – and the complexities and nuances of it defy the crude dichotomies we create around race, because it's so much more complicated. Somebody said to me, of Robert Cooper, 'How do you feel about him?' And I thought, 'He was a bastard, but he's my bastard.'

"The truth is, his blood is in my veins. He's the reason I have got lighter skin than some black people, and the reason my family have some privileges in contrast to some black families. There is no point disavowing the reality of who I am. I am this complex thing, and so was slavery."

It would seem that by taking a long-sighted and subtle view of history, Stuart sees the world not in black and white, but in a healthy shade of grey.

"With the book, what I didn't want was for my conclusion to be about blame. I wanted it to be about acknowledgement and remembering, because those were the things that came out of it for me. And the fact is, we hold in ourselves so many histories. Normally so much of the narrative around race is to try to make people feel bad, rather than to make them do better. And actually, I'd just prefer them to do better."

Sugar In The Blood is published by Portobello Books, £18.99. Andrea Stuart is at Aye Write! on April 20. For tickets go to www.ayewrite.com or call 0141 353 8000.