Stories have taught us that we need endings.

It's an idea so pervasive that it seeps into real life. We like to put a full stop to events. We want to write the last word. And often we'll leave it at that.

But real-life stories don't end, of course. And I'm beginning to wonder if we've forgotten that. Reading Rian Malan's book of journalism about South Africa over the past 20 years, I was reminded about Northern Ireland (the place I come from and a place Malan has written about). I thought about how Northern Ireland has rather slid out of the news in the past 15 years. In some ways, this is a good thing. It's a marker of the fact that the Northern Ireland of today is not the Northern Ireland I grew up in.

But history didn't stop on Good Friday 1998. In recent weeks we've had serious riots in a major British city and it's rarely been at the top of the news. Is this simply a reflection of the media's increasing Londoncentricity? A lack of resources on the ground? Or, most worryingly. a sense that the Northern Ireland story is now over? And this latest problem in Belfast just doesn't fit in with the familiar narrative and so is easier to downplay or even ignore?

The full stop won't be erased, in other words. You could say the same about South Africa. Indeed, Rian Malan has been saying it for the past two decades. Malan is a writer, a contrarian, a controversialist. A former leftie who has drifted to the right. Sort of. When he was younger, he ran away to America and became a music journalist using the pen name Nelson Mandela.

After he published his first book, My Traitor's Heart, a painful, mad, at times maddening, but always brilliant essay on South Africa and race, and the part his family played in both those narratives, he was branded a racist. At the same time he was also praised by the likes of Salman Rushdie and John Le Carre. He wrote about crime – black on white, white on black, and black on black – in Johannesburg and called it "the war that was and is and yet will be". My Traitor's Heart was a book that feared the worst about South Africa after apartheid.

And then the worst didn't happen – or not in the way that he thought it might. The new South Africa threw up new stories. The Lion Sleeps Tonight is a collection of the ones he has told over the past 20 years. Stories about Aids and the ANC, about violence and Winnie Mandela and music and politics. It is a book full of fear, frustration and surprises. It is a book ripe with horrors. But also – and you can feel the astonishment in his words when he says as much – strange new moments of hope. Reading it, I was annoyed, angered, amazed, irritated and enthralled, often on the same page. It's a book that loves rubbing liberal pieties in the dirt, that is keen to pick a fight. And yet he's too good a journalist to give in to mere polemic.

That has also got him into trouble. The most contentious piece in this collection, The Body Count, happened because he was commissioned by Rolling Stone to address then South African President Thabo Mbeki's claims that Aids was not the result of a virus but of poverty and bad nourishment. He jumped at the chance.

"I'd been struggling for years to get naive and idealistic Americans to publish anything even vaguely negative about the South African situation," he writes in his introduction to the piece. "Now they were offering me serious money to sever the presidential head and serve it up on an elegant literary platter. I said, 'Great, I'm your man'."

The problem was that while his research into the impact of the Aids virus on South Africa found it was hugely traumatic and deadly for far too many, he felt it didn't match up to the claims of a pandemic. Writing as much got him labelled a fool. He maintains that science supports his viewpoint. Research by Harvard University in 2008 suggested the Aids policies of Mbeki's government were directly responsible for the avoidable deaths of more than one-third of a million people in the country.

However, the thing is that there are so many other stories to tell in the new South Africa. In the essay that provides the collection with its title, Malan traces the history of the song The Lion Sleeps Tonight back to a Zulu songwriter named Solomon Linda.

It is, inevitably, a story about white writers claiming ownership of "traditional" tunes and then reaping the financial windfall. Linda died penniless in Johannesburg. Malan tells his story, the story of the song and its curious history in America. He also follows the money and asks whether some of it is going to find its way back to Linda's family.

"Following the money" is one of the continuing strands of this book as Malan teases out a vision of a new South Africa. The result of that vision is a thrilling, disputative, at times self-obsessed, book. It is also hugely alive. Malan's writing is animated by anger and a savage irony, yet it is always controlled, clear and readable. And, yes, his vision of South Africa is full of blood and fire. But it's the hope that sticks to you. The final essay, The Fabulous Alcock Boys, looks at two white brothers who can speak Zulu and are part of a new country, who may even, he suggests, be a new mutation of it.

In an afterword, Malan worries about rising post-apartheid ANC politicians who praise Mugabe and declaim whites as criminals. He worries that an ending, a bad ending, is imminent. "But the end has seemed imminent ever since I opened my eyes," he adds, "and somehow it hasn't arrived yet. Maybe it never will. Maybe we'll just mutate out of our present incarnations and become something else entirely."

Maybe. Not an ending then, but a beginning.