IT would be too much to say that Simon Callow's life has been Wagnerian, but there have certainly been moments of high drama. Like the time when his disciplinarian mother was beating him, not for the first time, with a broom and he snatched it from her hands and snapped it in two. That was the end of that. Or the moment he told his family he was gay and his aunt shouted, "Christ almighty!" and dropped her glass of whisky. Noise of breaking glass. Curtain falls. End of part one.

The reason the Wagnerian comparison suggests itself is because Wagner has always been one of the Callow's great passions and still is. The 68-year-old actor and writer remembers first hearing the music on his grandmother's hissy, crackly 78s and by the time he was an adolescent he was rather obsessed with the composer. Much later, he played Wagner on stage and has written a brisk and blustery biography of him, Being Wagner, which he will be talking about at the Edinburgh Book Festival this month. Callow says he didn't so much fall in love with Wagner's music as give in to the all-out assault.

Speaking from his London home, Callow says it may be significant that he first fell for Wagner when he was an adolescent. In his book, he quotes reviews comparing Wagner's music to a fungus or "something ejaculated by a masturbating pig", but as a teenager growing up in the 1960s, Callow remembers being drawn to the difficult, badly-behaved German composer for the same reason that, years later, teenagers were drawn to Kurt Cobain. "I think that's the territory that Wagner's in," he says. "Rockers would say the same thing: that all the rest of it is all just pretty, pretty rubbish – now we're getting to the visceral, the id. Both come out of the deep unconscious."

When Callow himself was discovering Wagner, in the 1960s, his id included some difficult areas. He was gay and hadn't told his family yet; he was also naturally solitary: he didn't have many friends at school, and was growing up in Chelsea in a very difficult, tense situation with a controlling mother whose theories on child-rearing were based mainly on violence rather than praise. "She always believed that any pleasure you took had to be repaid in work of some kind," he said. "There was no such thing as pleasure for its own sake."

The problem was, says Callow, that he and his mother were essentially incompatible, which led eventually to that melodramatic moment with the broom (when the broom wasn't to hand for a beating, his mother's favourite was a brush with steel prongs). As soon as Callow took the broom and snapped it in two, the beatings stopped, but that wasn't the end of the matter. "She then wrote to my school and said – This child is ungovernable, will you please give him six of the best immediately," he says.

Callow's escape was the relationship he had with his grandmothers, particularly his mother's mother, the one who introduced him to Wagner on the crackly 78s. "I was not a great one for friendship but I did have a wonderful relationship with these two grandmothers," he says. "My mother's mother was an extraordinarily engaging personality – she was a singer and she loved to be surrounded by music all the time. She also loved acting and I used to dress up. So I had that whole world and then later, every Friday night, I would go to them, where she and my aunt, who lived with her, would always have what they would call an 'evening'. They used to knock back the booze like mad and you were expected to perform."

When Callow decided to tell his family he was gay, it was to his maternal grandmother that he turned first. He had been at university for a year, studying English at Queen's University Belfast and had fallen for a couple of boys – he pined, and nothing happened. "That was where my confidence failed me," he says. "I was always confident in my ability to talk or discuss an idea but what I wasn't good at was putting myself forward as an object of desire."

Extraordinarily, he had already come out to some of his school friends – this was, remember, the 1960s: before legalisation, and a time when a confession of homosexuality would get you a punch rather than a hug. However, demonstrating a personality full of flashes of confidence as well as nerves, Callow just shrugged his shoulders and came out as gay.

"I think adolescent boys," says Callow, "certainly in my school in the early 1960s, when sex is in the air, it can take all kinds of forms and people would say to me, 'Who turns you on?' and they'd all be saying whoever the great pin-up of the moment was, and I'd say 'Cliff Richard'. Remember, I was in Chelsea and it was the swinging 60s, although the kids were mostly from Fulham and working-class areas. Everybody was grabbing each other's balls all the time and I dreamt of being seduced but that was never on the cards. Everybody knew I was gay and they were OK with it. I hadn't come out to my family, that was a different matter."

It was when Callow returned from university that he became much more conscious of the need to tell his family, but he wasn't sure how to go about it. At first, the secret of his sexuality had built up something of a wall between him and his grandmother and he hadn't spoken to her for four years. "I was aware that we were too close as well in a way," he says. "And then finally, I thought: I've got to see her. I went unannounced to the house and my aunt opened the door and said, 'You'd better come in'.

"My grandmother was thrilled to see me, but then she said, 'Have you got a girlfriend?' and I said, 'No, I don't'. And she said, 'Are you a homo?' And I said, 'Yes', and my aunt dropped her glass of whisky and said, 'Christ almighty!' My grandmother then, magnificently, just held my hand and said, 'Far be it from me to deny anyone love.' And that moves me even now."

Some 50 years on, Callow is now married to Sebastian Fox, a 34-year-old management consultant, having undergone something of a conversion from a sceptical stance on gay marriage to all-out support and his own wedding last year on the Greek island of Mykonos. "I wasn't a particular advocate of gay marriage," he says, "then I met Sebastian and it was quite clear that we could be together for the rest of our lives."

Callow is also aware, as well as slightly uplifted and bewildered, by the fact that gay marriage has happened rather quickly as part of a remarkable social, cultural and political transformation of gay life in Britain over the last 20 years. He notes that there are still pockets – large, spacious pockets – of homophobia in the world; Callow also says perpetual vigilance is needed in the face of homophobic bullying in schools and homophobic lyrics in rap songs. But on the whole, he is optimistic about the progress made and the potential for more.

I ask him if he recognises the part that his most famous acting role, as gay man Gareth in the 1994 romantic comedy Four Weddings And A Funeral, might have played in this cultural transformation, the slow and then quick-quick acceptance of gay rights, and he says yes. He recognised even at the time he was playing Gareth – who famously brings about the funeral of the title when he has a heart attack while dancing at one of the weddings – that it was a significant character.

"That was the beauty of it," he says, "because it was about a gay man who didn't die of Aids – it was in the middle of Aids, Four Weddings And A Funeral. That was the thing: he died of Scottish dancing. There should be a Government warning on the subject: Scottish country dancing can kill you! Neither Gareth nor Matthew were stereotypes of any kind – they were just original and interesting human beings who happened to love each other."

Callow has seen the representation of gay people on screen improve even more since Four Weddings, although he still thinks it's not enough. Where's the gay Romeo And Juliet or the gay Anthony And Cleopatra, he asks. He also thinks that there is a danger of the push for more gay visibility being lost in another debate entirely, one that touches on the pretty profound question of where gay people belong in society.

"In a way the whole issue of gay rights has been somewhat overwhelmed by gender identity, which has sort of pushed it out of the way a bit," he says. "That's become the big drama of the age, although in fact, it affects, I can't help feeling, quite a small number of people. All this LGBTQZ hyphen asterisk ... it's an attempt to create a continuum of sexual identity which doesn't exist at all – there's no reason why it should. There's nothing in it apart from the fact that it's about different experiences."

Callow is aware that could sound like the Jurassic views of a man of a certain age, and he admits that there is a certain part of him that pines for how things were for gay men long ago – he particularly remembers the gay pubs and dives of Edinburgh in the 1970s and their spirit of conspiracy and solidarity, and, although it's changed and cleaned up, is looking forward to coming back to the city for his date at the book festival.

One of the questions he will be tackling at the event is the tricky one of whether Wagner deserves admiration at all. Not only does Callow's book acknowledge that Wagner was often insufferable, petulant and difficult, it also tackles the question of his poisonous anti-Semitism. Is it OK to love his music knowing the truth about his views? If you listen to a beautiful piece of music, or read a wonderful poem, and then discover that the creator of that work of art was a horrible bigot or racist, does it matter?

Callow's view is that you must make a judgement on the work, and separate the creator. He cites the example of Eric Gill, who abused his daughters but also created the statues of Prospero and Ariel at the front of BBC Broadcasting House in London. What's to be done with the sculpture, asks Callow. "Is it to be chipped away, taken down, covered up? Because in itself there is nothing but beauty." He thinks the same of Wagner: his anti-Semitic views were poisonous and hateful, but there is nothing anti-Semitic in the music itself. Yes, to some, the music is intolerable, but for Callow there is a profound, disturbing pleasure to be had from it. It goes deep into the human pysche, he says, and his challenge to the rest of us is: dare to listen, dare to look.

Simon Callow is at the Edinburgh Book Festival this Friday, August 18 at 6.45pm www.edbookfest.co.uk His book Being Wagner: The Triumph Of The Will is published by William Collins