One lover described him as too good-looking for a man.

His dad called him "the horizontal champ" for the way he used to lie in front of the fire like an exotic cat. Ask anyone about Terence Stamp and before you can say face of the sixties, the Terry who met Julie at Waterloo Station every Friday night, beau of Jean Shrimpton, sweet transsexual in Priscilla, Queen Of The Desert, or Steven Soderbergh's London gangster in The Limey, the word "beautiful" comes up. Never handsome, always beautiful.

"He'll be the first to admit how beautiful he still is," says Paul Andrew Williams, the director who has taken the looker from Bow and turned him into a grumpy old man in the new British comedy-drama, Song For Marion. With Vanessa Redgrave as his wife, the pair play an ordinary, elderly couple living on a council estate. Marion is ill and keeps her spirits up by singing in the local choir, Arthur is terminally grouchy and terrified about losing the love of his life. Stamp may not know much about being ordinary, but he's a Nobel prize winner when it comes to broken hearts. Stamp and Shrimpton were the Posh and Becks of their day, Burton and Taylor without the bling.

Famously choosy about the parts he plays ("I don't like to do crap unless I haven't got the rent," he writes in his memoirs), Stamp, now 74, thought Williams had written a wonderful script. Having seen one of his previous films, the thriller London To Brighton, and had a look at the rest of the cast list, which includes Gemma Arterton as the choir mistress and Christopher Eccleston as Arthur and Marion's son, Stamp knew Song For Marion was going to be a cut above. But that B word, beautiful, kept nagging at him.

"How can I say this without seeming unusually vain -" he begins. There is a long pause. It is going to be the first of many as we take tea on the rooftop terrace of a hotel during the London Film Festival. Stamp, who has devoted a fair part of his life to seeking enlightenment from the East, is a man who is very comfortable with silence. He takes his time to get just the right answer. Waiting for him to do so would normally be torture for an interviewer facing a ticking clock, but the answers are worth waiting for (not always the case with actors), and, in Stamp's case, let's just say the view while you wait is not too shabby.

While considering whether he should do Song For Marion, a friend told him there was only one problem: everyone would know he was a pensioner, and once that door was opened it could not be closed again. "He knew I don't see myself like that."

Nor do most people. Courtesy of his films and photographs, some of the most remarkable taken by David Bailey, Stamp's beauty is a matter of record, like parliamentary debates and court rulings. Michael Caine, with whom Stamp shared a flat in the early, hungry days, told him the camera was "his lady" and to never forget that. He should have added that while the lady never ages, those she gazes upon always do, eventually.

Michael Caine, David Bailey, Julie Christie, Jean Shrimpton, Federico Fellini, Marlon Brando (his Superman II co-star), Princess Diana ("her company was heaven"), Bob Dylan - like Woody Allen's Zelig, Stamp has collected famous friends, co-stars, lovers and acquaintances like, well, stamps. It is not bad going for the son of a tugboat captain from the East End of London, as he would be the first to admit.

Born in 1938 to Tom and Ethel, Stamp was the first of five children, one of whom, Chris – manager of The Who – died last November at the age of 70. Hearing him speak about his mother and father, and the way he writes about them, it is clear that he saw something of his own parents in Arthur and Marion. Like them, they were "twin souls" who had found each other in the maelstrom of life.

There is certainly something of his father in Arthur, a working-class man who loves his son dearly but would never say it openly. We start to speak about his father when discussing his Song For Marion costume. Clothes are to Stamp what oxygen is to the rest of us. They matter. A lot. In the case of Arthur's character, it was the Clarks desert boots that proved to be the madeleine.

When he was younger, Clarks dezzies were the very dab, he recalls, but you could only get them in a few places, Glasgow being one of them. He got his first pair when he started earning as an actor.

"My dad was very elegant, very poor but very elegant. My brothers and myself, those of us who had taste, got it from my dad. He just had it. He didn't have money but he had style. Part of the great pleasure of my success was getting him things he could never have afforded. Even when I was not well off I managed to get him a pair of Clarks."

Dad was good looking and funny. He could have had any woman he wanted, says Stamp. "But he only ever loved my mother. What he gave up was extraordinary, really, in order to keep her. Like, she wanted kids; he would never have wanted kids. He was like me, a loner. So he sacrificed. But what he got was this love of his life. He was never unfaithful. He was a drinker but every Friday he brought home the wages. I thought, that's like a twin soul relationship."

Stamp was not as close to his dad as he was to his mum. It was the way of the times. Stamp senior had gone to sea when he was 15. He grew up in a tough, all male environment where it wasn't the done thing to reveal your feelings. "He was very funny but rather wicked funny. He was only really social in the pub – I don't remember anybody coming into our house, no visitors."

Terence was his mother's son. She never wanted him to leave home. It was her death in 1986, while he was in New York filming Legal Eagles with Robert Redford, that started him writing. He wrote her a letter and set fire to it in Central Park ("a gesture I felt she might appreciate") and he hasn't stopped writing since, producing three volumes of memoirs, a novel and even a cookbook. The memoirs are funny, tender and wise, like Rupert Everett minus the bitchiness. Be warned, however: the reader has to endure a fair bit of Eastern mysticism and actorly musings about craft along the way. He also has a thing for star signs.

Stamp had his own "twin souls" experience once, and it was not, alas, with Elizabeth O'Rourke, whom he married in 2002. She was a former pharmacy student and 28, he was 64. His first marriage, it lasted six years. "She just got bored with me," he says. "People find that hard to believe, how did she get bored? She got bored! The kind of life that I was leading, after the thrill of the first few years - This is me giving her an opinion. I have never really spoken to her about it. I realised that this was not how she envisaged it."

His twin soul was the Shrimp, Jean Shrimpton, the original supermodel, even more super than Twiggy. When he first met her she was with David Bailey, and her beauty made him gasp every time he saw her. My God he adored her. He loved her so much he became terrified of losing her. When she briefly left him his fears became real. "Unable to contemplate life without her, I pushed her away," he wrote in Double Feature.

He fell into a deep depression, complete with suicidal thoughts. He got high. At one point he lay down and willed himself to die, like an animal. He picked up a couple of hitchhikers who then pulled a gun on him. Such was his mood of despair he told them to "pull the trigger or piss off". They ran from the car.

Looking back today, he realises he was just young and careless, careless about other people's feelings. He believed she would love him for ever. "I thought it was always going to be like that, I didn't realise that was it."

He can even say now that her ending the relationship was "probably" the best thing that happened to him. The way he tells it, his life was a ship that left Southampton bound for Shetland, but due to a tweak on the compass, he wound up in Reykjavik. (Since we've got the atlas open, I should say that he now lives "on the move" between London and the US.) "That's what happened to me. I wound up in Reykjavik because [Jean leaving] was such a shock. It proved to be such a shock to me that I began to view my life differently."

He went travelling, to India, Egypt, Japan and Ibiza (to help on a friend's organic farm), and sought enlightenment from wise men wherever he could find them. In one case it was the guru Jiddu Krishnamurti, in another it was Fellini, the director who cast him in 1968's Spirits Of The Dead and pulled him out of the post-Shrimp slump. Wherever he has gone, whatever he has done, from working on his 1962 breakthrough film Billy Budd, directed by Peter Ustinov, or with Soderbergh in 1999's The Limey, he is always asking questions and seeking advice. Perhaps that's why people are forever finding him beautiful. By fixing them with those dazzling eyes, and being interested in them, he makes his subject feel like the most fascinating thing in the room. They see themselves in him, like a mirror, and like what they see. Beautiful people can do that.

When he was the face of the sixties fame had its pleasures, and plenty of them. No restaurant was ever fully booked if Tel turned up. Tailors struck oil when he walked in the door. His ex-wife once said he knew more about clothes than acting. Today, for fellow dedicated followers of fashion out there, he is wearing a corduroy suit the colour of runny honey, a blue and white striped shirt that brings out the azure in his eyes, and handmade shoes. He tells me the dates when everything was bought: 1969, 1968, the suit he acquired for a movie. He buys things to last. Comes from once having nothing, he says. His dad was the same.

It was his dad who, seeing young Terence's fascination with actors when the family got its first television, told him: "Son, people like us don't do things like that." But he did, and after Billy Budd, for which he received an Oscar nomination, he was phenomenally successful, even if he was sometimes a lousy picker of parts, leaving Alfie to Caine, Georgie Girl to Alan Bates, and Camelot to Richard Harris. He became what he calls one of the "young, educated, working-class tigers let loose on the world, and on showbiz".

There was still the sense of something missing, though. Although he had been a grammar school boy, he left school feeling he hadn't learned very much. "I was a kind of a conundrum. I wasn't stupid but I appeared to be stupid because I couldn't learn by rote. So everybody just assumed I was thick."

Fame bought him two things: the confidence and means to carry on acting (to eat well, to look good), and the money to buy books and other beautiful objects. He had an eye, or when he didn't he had a friend who did. It was the books in particular, more than the chichi restaurants or other trappings, that gave him the biggest kick. "I could study anything. That's what I did."

He has made fortunes and lost them, most of the latter being done in his "resting" and travelling years when he couldn't get work or didn't fancy what was offered. His comeback came with 1994's The Adventures Of Priscilla, Queen Of The Desert, in which he played Bernadette, a transsexual hauling herself, with two drag queens, across Australia. He says he looked like "an old tomcat", but critics and audiences alike warmed to the comedy. The Washington Post said he looked like "Marlene D with killer eyes".

The Limey, in which he played an East End geezer coming to avenge the death of his daughter in LA, brought him to a new audience. The likes of Wanted (with James McAvoy), Yes Man (Jim Carrey) and The Adjustment Bureau (Matt Damon) followed. The old hipster had become hip again. The face of the sixties had made it to the noughties.

And now he's donning an 'orrible old raincoat and a scowl in Song For Marion. It is a risk in some ways. For the first time in a while, the "horizontal champ" is standing up and asking to be counted more for his acting than his looks. He even sings, something he has long been reluctant to do on screen. He is not worried, he says, but he is curious as to how people will respond.

When he looks in the mirror in the morning, what does he see? A figure that's ageing, he says, but that doesn't chime with what he feels is the reality. It will be terrible, he says, if he stays young here – he points to his head – but his body won't work properly.

It comes to us all, I offer. Age, the great leveller. "Of course it does, but it's very in focus with me because there's no sort of retirement, as it were. Things keep coming up and I keep engaging."

In The Limey, Stamp starred alongside Peter Fonda, another young tiger of the sixties. In one scene, Fonda's young girlfriend is lying in the bath asking him questions about all that ancient history. "Must have been a time, huh?" she says. "A golden moment."

For Stamp, it was. And for Stamp, though older, the golden years go on. n

Song For Marion (PG) opens on February 22.