Behind the beat: how On The Road made it to the big screen

The well-groomed and conspicuously white audience is listening to a black bebop band, three elegant and restrained musicians led by a larger-than-life singer in a wide-shouldered, double-breasted suit and spats, whose performance style involves comically contortionist dance moves and bizarre lyrics – the combination making him seem quite demented. The excited response is magnified at the back of the room, where two young, exceedingly handsome men – under-dressed in borrowed jackets and no ties, and as alien to the room as the band itself – prop up the bar and sing along to the outlandish song, Yep-Roc Heresay. "Slim knows time!" yells one of them. And it's clear they do too.

Having finished the number, the singer stats to flirt with a young woman in the audience, offering to "see you there in the back room, baby". Suddenly, from the shadows at the edge of the room, the Brazilian director Walter Salles calls "cut".

The spell broken (but only just), I'm back in the present, and the film set of Salles's long-awaited adaptation of Jack Kerouac's classic Beat novel, On The Road. To be precise, I'm in Montreal, one of the film industry's most affordable, all-purpose locations, here standing in for San Francisco circa the late 1940s. It's a real club, in Old Montreal, but one that has been magically transformed to shoot us into the past.

Kerouac's semi-autobiographical tale recounted his formative travels around the United States, for the most part criss-crossing the country in a speeding Hudson Hornet with his wild-living friend and muse, Neal Cassady. In a Cold War era when the common advice to young people was to stay at home, behave, and beware of Reds under the bed, Kerouac and Cassady – under the guises of Sal Paradise and Dean Moriarty – represented a restless desire for new experience and defiant self-expression. Kerouac's book sowed the seed and coined the term for the radicals and iconoclasts who would follow him, "rising from the underground, the sordid hipsters of America, a new beat generation".

The scene I've been watching is one of a number in the film which illustrate the Beats' love of jazz, the improvisational nature of which heavily influenced Kerouac's writing. On stage is the film's version of real-life jazz curiosity Slim Gaillard, a man whose lyrics included many words he simply made up, and brought to life by the fittingly eccentric Coati Mundi of Kid Creole And The Coconuts; the aficionados at the back of the room are Paradise and Moriarty, played by actors Sam Riley and Garrett Hedlund.

During a break in filming (Mundi needs some adjustments to his bouffant hairpiece) Riley comes over for a chat. "Crash, crash, potato chips," he says, mimicking the singer. "It's pretty bonkers, isn't it?" He's a charismatic chap, imposingly tall and with brown eyes that don't let you out of their grasp, matched by a distinctive smoky timbre to his voice. He also has a Yorkshire accent. The 32-year-old from Leeds certainly doesn't fit one's idea of the French-Canadian, Boston-raised Kerouac.

Riley already knows a little about taking on celebrated characters, both real and fictional: he burst on to the scene with his terrific performance as Joy Division's ill-fated singer Ian Curtis, in Control, then dared to reprise Dickie Attenborough's scarred villain Pinky in a remake of Brighton Rock. But he wasn't prepared for a Kerouac expert he met during preparation for On The Road.

"I was introduced to this guy, and you could see the shock and horror on his face. 'You English?' Yep. 'You're awful tall to play Jack.' Yeah. 'You realise he was impossible to tackle on the American football field? That he had piercing blue eyes?' I had a bit of a laugh with him and pointed out that I smoked the same brand of cigarettes, Camel filters, so 'one out of five'. But it was then that I thought, 'Oh s***, this is going to come under a lot of scrutiny.'

"It was exactly the same playing Curtis. When we were shooting the concert scenes there were people – extras they'd got from the Joy Division fan site – coming up and saying the most terrifying things to me. But what idiot would turn down this part just because he's British? I guess I like sticking my head out."

His encounter with the Kerouac biographer took place during the Beat "boot camp" Salles set up in Montreal before the shoot, to enable his actors to immerse themselves in the world of their subjects. They talked to academics and family members of the book's protagonists, watched a documentary Salles had made about the Beats (as part of his own, typically rigorous research) as well as jazz documentaries and feature films from the period.

Riley was particularly impressed by Kent MacKenzie's 1961 feature The Exiles, which follows a group of native Americans as they drift through Los Angeles. "Walter wanted us to absorb the pace, the urgency of young people at the time, which maybe is lacking nowadays. The guys in The Exiles have this amazing swagger. By the end of the boot camp we were all chomping at the bit."

As for Kerouac, the ebullient actor points out that he's not impersonating the writer, but playing his alter ego, Paradise. That said, the whole cast – from Viggo Mortensen, who plays Old Bull Lee (William Burroughs) to Kristen Stewart, whose Marylou is based on Cassady's first wife Luanne Henderson – has steeped itself in the real-life templates for the book's characters. "I'm trying to base as much of the background on Kerouac himself," says Riley. "I got myself fit – because he was an athlete and I was a streak of piss – and did dialect sessions based around the recordings I have of him. Kerouac's accent is beautiful, somewhere between New York and Boston and wherever he feels like stealing from. He was a sponge for experience, in love with the country and the way people speak in it."

As the production has taken him to New Orleans, Arizona, San Francisco and Mexico, Riley has also fallen in love with North America. "I've discovered this varied, fascinating, wondrous continent, and I was experiencing it in the original story about life on the road. I think we're all pinching ourselves, actually."

At the same time, he was unprepared for the all-encompassing nature of a Salles shoot. This is a director who followed every bike track and footstep taken by Che Guevara around South America for The Motorcycle Diaries, and whose current production is running at some 50,000 miles, five months and counting.

"This is nothing like anything I've ever done – work, or life-wise really. I've never been this long away from home, or family. I've never travelled so much in such a short amount of time, or had to give so much. It has been very intense."

Riley laughs at the recollection that he and his wife (German actor Alexandra Maria Lara) had to spend their first anniversary in Montreal. "It's a hell of a lot to ask of a newly married man."

The genial Hedlund comes over to say hello, with an expression that might be described as halfway between Cheshire cat grin and crocodile smile – befitting a character who is a dreamer, a conman and an irresistible, if unreliable lover who sleeps with four people in the film.

"I've had two years of history with Sam, since we first auditioned together. I knew he would be my brother," he says warmly. "I was so eager for him to do this, because there isn't any characteristic that is false about him, in terms of his bohemian abilities and tendencies." And then he strolls off, leaving me to wonder what those might be.

An hour later the shoot has moved outside, into biting wind and rain, for a long tracking shot of Sal and Dean walking down a San Francisco street. The road is lined with period cars – big, curvaceously bonneted monsters – and the shop fronts aglow with nocturnal activity. While dozens of production people are hidden inside woolly hats and thermal coats, the actors are in their T-shirts, feigning the easy stride of a sultry summer night. The Midwesterner and the Yorkshireman make it look easy.

Kerouac wrote On The Road in 1951, in a three-day outpouring of "spontaneous prose" – some say fuelled by Benzedrine – typed on to a single 120ft roll of paper, the legendary "scroll". The book then underwent years of revisions with nervous editors before being published in 1957, at which point Kerouac became the reluctant figurehead of a literary movement whose other seminal texts include Allen Ginsberg's Howl and Burroughs's The Naked Lunch.

The film's gestation has been even more protracted – taking more than 30 years. Francis Ford Coppola, the renowned director of The Godfather movies and Apocalypse Now, bought the rights to the book in 1979. In the years that followed different directors, scripts and stars – including Dennis Hopper, Brad Pitt and Colin Farrell – came into the frame, yet never in the right combinations to make film financiers part with their cash.

On The Road was gathering dust on the shelf when in 2004 Coppola saw The Motorcycle Diaries and finally found the man to help him realise his dream. Not that Salles has had an easy ride himself. American companies have remained unmoved by one of their country's iconic tales, leaving it to the French to stump up the cash. When Salles calls the production "an odyssey", one assumes he means to convey Homeric agony as much as mere distance.

But if anyone can keep a team together through thick and thin it would be the charming, passionate, committed Brazilian director. These attributes also explain an enticing cast, the director surrounding his little-known leads with old hands like Mortensen and Steve Buscemi, and a wonderful group of female actors – Stewart, Kirsten Dunst, Amy Adams, Elizabeth Moss and the Brazilian Alice Braga – playing the women he calls the "silent heroines" of the piece.

For their principal source, Salles and scriptwriter Jose Rivera (who was Oscar-nominated for The Motorcycle Diaries) decided to draw on the original scroll, rather than the edited book, because of its more overtly autobiographical nature – including Kerouac's loss of his father, which is in part what propelled him on to the road – and bolder expression of the story's sexual aspects, including Moriarty's bisexuality.

But he also observes that On The Road is "an extraordinary blend of experience and imagination – Kerouac riffed on what he had lived – and we wanted to reflect that". One of his favourite examples involves Burroughs' house in Algiers, a suburb of New Orleans.

"We found it during our research, because it was one of only two houses in its street still standing after Katrina. People say that if a house could survive Burroughs, it could survive anything. And it is nothing like the rambling old house described in the book. But it's Kerouac's invention that we have decided to honour."

As well as the celebratory nature of the story, Salles is keen to explore the darker themes, not least the tragedy of Cassady, who would be just 42 when he died, having passed out alongside a railway track.

"Dean was described as the western wind, the kid who spent a third of his life in jail, a third in the pool hall, a third in the public library. This almost mythological character is also the cowboy crashing at the end. So his journey is a tragic one. This is a story about the end of the American Dream, and about a broken friendship. But even if that friendship is over, it is immortalised in the book. So yes, the pain is there, but also the possibility to transcend it through art."

The next day the crew are shooting an early scene in a Colombia University bar, featuring Sal, the Ginsberg-inspired Carlo Marx and others; the wood-panelled booths, bow ties and sweaters create a preppy air. The scene opens with Sal bemoaning his writer's block, and ends with the announcement that the infamous Cassady, "the jailhouse friend from Denver", has arrived in New York.

Ginsberg is played by another up-and-coming Brit, Tom Sturridge, a darkly intense-looking young man who will be giving a version of the poet, just 19 at the time, far removed from the balding, big-bearded Buddhist with whom most are familiar.

"Kerouac and all the people writing about Ginsberg in his youth describe him as a weird, wounded animal," Sturridge tells me. "He had an incredibly complicated youth. The discovery of his sexuality was a slow and difficult process, which Neal Cassady opened up but also completely f***** up. His relationship with himself was one of disgust and yet a kind of incredible, prophetic arrogance; somehow he knew he was going to be someone important."

Like everyone else on the film, Sturridge hasn't stinted on his homework, having read the poems, diaries, letters, "everything" ever written by his character, and which is proving to be most useful when the actors are invited to improvise.

"It's not like you're doing a film about three normal teenagers in London, and can say any old rubbish," he laughs. "These guys were incredibly well read. Their frame of reference is significantly different to mine and Sam's – you've got to arm yourself. So the night before a scene I'll look over the script, go over the diaries, go over the letters, find stuff that you bring out spontaneously."

Riley recalls his scenes with Mortensen in New Orleans. "I think he has wanted to play Burroughs for a while. He turned up with his own pistol and shoulder holster, his own old books – by [Louis-Ferdinand] Celine, for example, who Burroughs really admired – that we could use in the movie because they were first editions. And he had a million ideas. He was inspiring.

"But I was pretty nervous about the improvisation. Often we've come to the end of a scene and there's no cut, we just keep going, shouting out whatever we feel, within context. I was anxious about what Viggo might start staying. 'So what do you think about Nietzsche, Sal?' 'Ah - Ubermensch?!'"

Interestingly, it's not just the elder statesmen of the film, Salles and Mortensen, who grew up with the novel. Sturridge first dipped into it when he was 13, reading it properly when he was 20. "It's one of those seminal novels, on a literary and a human level," he suggests, "something that we all have a connection with, whether we love it or hate it."

And refuting the notion that it's only one for the boys, or old hippies, one of the most vocal fans is the 22-year-old Kristen Stewart. Months later, at the film's Cannes premiere, she will speak of how On The Road changed her life.

"I chose the book from a school reading list in my freshman year, because it looked much more fun than any of the others. I knew that it was about counter-culture, and when you're 14 years old and putting ridiculous anarchy signs on your back pack, that's what you're drawn to.

"I come from a very comfortable, very fortunate household, but the kind where you can become lazy and complacent. So when I read On The Road I thought, 'I have to find people like this, who will push me, who don't compromise on their desires, even if they're different to the norm.' It really did inform how I wanted to live."

For those who question whether Kerouac's celebrated work has something to say to today's young people, Stewart was smilingly assured. "I don't think the counter culture has ever gone away."

On The Road (15) is released on October 12.