One night in 1953, a young singer took to the stage of New York's legendary jazz club The Village Vanguard.

Harlem-born but raised by his grandmother in her native Jamaica, he performed a selection of Caribbean folk songs and traditional calypsos. His name was Harry Belafonte.

For the 26-year-old the Vanguard gig was a regular slot, earning him around $70 a month which he used to pay for acting classes. This particular night was different, however, because in the audience was one John Murray Anderson: born in Newfoundland to a father from Saltcoats, and educated at one of Scotland's top public schools, Edinburgh Academy, he was among Broadway's most successful impresarios.

Anderson had directed the young Bing Crosby in his first film, The King Of Jazz, and he knew talent when he saw it. In Harry Belafonte he found it in abundance. "I've selected you to be in my revue," he told the singer when he came off the tiny stage that night. "That means you're the best there is."

Anderson picked Belafonte for a leading role in the John Murray Anderson Almanac, a Broadway revue which opened in late 1953 and fast became a hit. It won the struggling actor a Tony award and one of the songs he wrote for it, showstopper Mark Twain, would later form the centrepiece of his first album for RCA, Mark Twain And Other Folk Favourites.

The revue also brought Belafonte to the notice of Hollywood director Otto Preminger, who cast him in his 1954 musical Carmen Jones. By 1956, Belafonte was a mainstream recording star and by 1959 he had added an Emmy to his trophy haul, becoming the first African-American to win one.

Anderson, however, was dead, killed by a heart attack the year after his Almanac revue opened on Broadway. The Scot missed out on seeing his protege rise to become one of the most successful black performers of his generation and he never witnessed the other great role performed by the man he had "selected" that night in the Greenwich Village jazz club: that of social agitator, political campaigner and luminary of the civil rights movement.

Now 85, it's that part for which Harry Belafonte is most revered. But in a new memoir and documentary celebrating his life and work he recalls that fateful meeting with the flamboyant Scot – a man he calls "a legend" – and the galvanising effect it had on his career.

"He'd started out in vaudeville, gone on to produce the Ziegfield Follies along with Billy Roses's revues, done movies, and even overseen the Ringling Brothers' circus," he writes in My Song. "An effervescent figure, always just in from Paris and making a grand entrance with two or three theatrical grand dames, Anderson had an air of infallibility about him."

Belafonte's autobiography is published tomorrow. The following day sees the release of Sing Your Song, a biographical documentary by New York-based film-maker Susanne Rostock. There's also a soundtrack album which features Mark Twain as well as other favourites from the Belafonte canon, such as Jamaica Farewell and Banana Boat Song (Day-O).

Rostock, a softly-spoken 63-year-old, spent two years interviewing Belafonte, often over green tea and pumpkin muffins in her editing suite. Though she first encountered him 60 years after John Murray Anderson did, she was equally struck by the force of his charisma.

"My first impression was this is probably the most powerful person I've ever met," she says simply. "He just commands the space."

Belafonte's Broadway years, his Hollywood films, his reign as the "King of Calypso" in the 1950s and his trailblazing career as a black host of primetime TV shows in the 1960s are an important part of Rostock's story. But the film's heft comes from its exploration of the times the singer lived through and in particular his strenuous opposition to segregation and racism. And don't think his fame insulated him from either: as late as 1968, the corporate sponsors of a TV special Belafonte made with British singer Petula Clark threatened to pull the show after she touched his arm during a song.

"Originally I wanted to call the film This Is Not Your Mother's Harry Belafonte," Rostock laughs. "It was a joke, but only a half joke. Because whenever you speak to anybody about Harry Belafonte they say 'Oh my parents used to love his music'. So it was always through his ballads and his calypso music that people knew him. His political activism was very much under the radar and behind the scenes. It was never something which was loudly publicised."

What's notable throughout the film is the way Belafonte appears, Zelig-like, at the side of so many of the era's defining historical figures, people like John F Kennedy, his brother Robert Kennedy and, of course, Martin Luther King, who had a profound influence on the singer.

In this, Sing Your Song is a triumph of the archivist's art. In the US National Archives, for instance, Rostock found a DVD labelled simply "Harry Belafonte" which turned out to be footage of a discussion about civil rights from a CBS special featuring Belafonte, his great friend Marlon Brando, Charlton Heston, the author James Baldwin and the film director Joseph L Mankiewicz.

In another extraordinary cache discovered in the basement archive of broadcaster ABC, Rostock found images of an outdoor concert Belafonte organised in March 1965. "It looked like a cat had used it as a scratching post," she says. "But I had the sound on a different track so I could hear it and because of the wonderful new digital techniques I could work it up – and there it was."

Featuring Nina Simone and Joan Baez among others, the concert was held the night before the 25,000 participants on Martin Luther King's famous four-day march from Selma to Montgomery march entered the Alabama state capital. It was this same march which turned Rostock, then 15, on to politics. "That's the only footage that exists," she says. "It's incredible".

The same word could be applied to Harry Belafonte's life. Sing Your Song sometimes approaches hagiography, but looking back across the span of that life it's easy to understand why.

Not that he is a man without regrets. He admits he often pursued his political activism at the expense of his family and is clear-sighted enough about today's America to know that the advances of the civil rights era have long since stalled.

"Somewhere we blinked," is how he puts it towards the end of the film. "We dropped the ball and we shouldn't be where we are. We should be further along'."

Sing Your Song is released on Friday and screens at Edinburgh's Filmhouse. It will be shown on BBC2 later this month. My Song: A Memoir Of Art, Race And Defiance is published tomorrow (Canongate, £14.99)