Fame, riches, women ... Neil Diamond has had it all, but it took an album about loss to make him cool
By Craig McLean

Here's a great Neil Diamond moment. It's the late 1980s, he's at the White House, and he's dancing with Diana, Princess Of Wales. "She was a wonderful dancer," he recalls. "She was very very human. I liked her immediately. She was a regular person, not born and bred to be the queen of England. A real person. She told me that she tried to go and see my show at Woburn Abbey. She was about 16 years old, and her dad wouldn't let her go. I guess he didn't want her associating with the rock'n'roll rabble that he heard about. In my case," he twinkles with dry humour, "my audiences."

Another Diamond moment. It's last November and he is performing, via satellite, at the 50th birthday party of Caroline Kennedy Schlossberg. He uses the occasion to finally admit that his 1969 hit Sweet Caroline - a song covered by both Elvis Presley and Frank Sinatra - was inspired by the daughter of John F Kennedy. "I saw a photograph of her when she was a little girl - she was walking or riding a pony. I just thought it was sweet picture, so I wrote the title down." He finished the song years later, in a Memphis hotel room.

Diamond has been close the Kennedys for years. In 1972 he performed at a political fundraiser for George McGovern. Ethel Kennedy, widow of Robert Kennedy and one of the event's hosts, asked Diamond to sing New York Boy. Instead he sang Sweet Caroline. Ethel promptly poured beer over his head.

"It was definitely a mark of affection," he says with a chuckle. "They like high-jinks and I didn't do the song she wanted me to do, and I got paid back for it. Are the Kennedys still a potent force in American politics? I don't know. I suppose if they were, Ted Kennedy would be president and Caroline Kennedy would be a senator at this point. But they do symbolise times that we were united as a country, and certainly President Kennedy was a great uniter. He was greatly loved in this country."

And another moment. It's 42 years this month since Neil Diamond had his first hit, with Solitary Man. In some ways it's Diamond's defining tune. He might be one of the greatest showmen in music history, writer of anthems that light up radio stations and arenas the world over, while wearing shirts that do the same. He's a pretty great uniter himself. But he is a solitary person. Melancholy and reflection hang over him. He talks with a slow rumble, and not just because he's 67.

Many years after Solitary Man reached number 55 in the US charts, it would become the title track on one of Johnny Cash's valedictory albums. It's a version in which the pain of Cash's final years looms large and uncomfortably. "That's a difficult record to listen to. But I think it made the performance even more powerful, the fact that this man was going through this terrible part of his life.

"The fact that the song is still around, and part of the vocabulary of music, is a happy thought. You never expect it. When I was writing these songs, they were for singles, they came out, they were around for three months if they were successful. And they were gone. Then the next single came out. You didn't really think about the longevity of the songs. So I'm happy to hear my early songs recreated - and recreated by artists today, whether it's Johnny Cash or anybody else."

And check out this moment, right now: 120 million album sales into his career, Neil Diamond is about to release his 27th album, Home Before Dark. It was produced by Rick Rubin: a legendary music industry figure who mentored that final run of Johnny Cash albums (and introduced Cash to Solitary Man), and has also worked on mega-selling albums by Dixie Chicks, Justin Timberlake and Slayer. He produced Diamond's last album too, 2005's 12 Songs - a simple title for a simple collection, a dozen numbers of acoustic introspection, a million miles from the sequinned showbiz world that Diamond had come to inhabit from the mid-1970s onwards. 12 Songs gave him the best chart placing of his career, and introduced him to a new, younger, audience. Diamond doesn't give a monkey's about being credible or "cool" - never has done; "popular" was what he was after - but now, at last, in his pensionable years, he was both.

Now Diamond and Rubin are following 12 Songs with the similarly minimal, equally powerful Home Before Dark. There's an accompanying world tour as well, Diamond's first since 2005; that year, only The Eagles and U2 toured more profitably. But that solitary man is still there - album songs such as the title track and If I Don't See You Again bring out Diamond's inner loneliness.

"I think most of my songs are tinged with that, although they're all positive. It's certainly not hopeless. But yeah, there's always been a tinge of loneliness in the music."

Why? Diamond doesn't really know. He had a loving childhood with his Jewish immigrant parents in New York. His two marriages ended amicably, he has a long-term girlfriend, and he has four children and five grandchildren. But Rubin coaxes soul-deep honesty out of artists. Certainly with Cash he encouraged the ailing legend to face up to frailty and mortality (his classic version of Nine Inch Nails' Hurt, complete with beautiful but agonising video, being a case in point). Is this what he does with Diamond?

"Uh, no, I don't think so," Diamond says with quiet finality. In conversation he's not a man for melodrama or over-statement. "But he does allow his artists to just let their mind go and not focus in on any particular thing. Let the mind be free, find a destination and write about it."

Neil Diamond is giving me a tour of his Los Angeles recording and rehearsal studio, Archangel. We had finished the interview and I was already walking down the street towards west Hollywood when he had an assistant run after me. Did I want to stay and have a look around? What a nice man.

He bought this place in the mid-1970s, a few years after he moved to the west coast from New York. The walls of Archangel are covered with memorabilia from Diamond's stellar career: gold and platinum discs, tour posters, caricature portraits from magazines. There are images from Jonathan Livingstone Seagull, his soundtrack for the film of the cult 1970s book ("I had to figure out how a seagull thinks!"), and from The Jazz Singer, the 1980 film in which he took the title role and acted alongside Laurence Olivier. Such was Diamond's standing back then that he was paid the then highest ever fee for a first-time actor. The movie wasn't a huge hit, but his soundtrack spawned another three big Diamond smashes: Love On The Rocks, Hello Again and America.

It's all a long way from his very first "gig", at his own bar mitzvah. Does that count? He grins. "It counts. But you don't do it as a singer, it's really a recitation of a prayer. I put my whole being into doing that song. Physically I moved the way I saw the elders of the synagogue move. You bow every time you say the word adonai - which means God in Hebrew - and I didn't know that. I saw them bowing as they went along so I bowed up and down for the whole song, the whole speech. But I looked forward to it and it was fun and I wasn't nervous particularly. I'd say it was my first public performance."

But it wasn't always just about music. In his teens he was an enthusiastic fencer - years later he even began to write a ballet about the sport, but eventually (ahem) couldn't see the point. "I loved fencing, but it wasn't a deep thing. I wasn't gonna pour my heart into songs about it." And, like a good Jewish boy, Diamond seemed destined for a proper career - on leaving school he took up a fencing scholarship at New York University, intent on following a career in medicine.

"That was my first immature stab at seeing myself and how I fit into society. I was good at the sciences in school. I thought of being a doctor, then my grandmother, who I was very close to, came down with cancer and eventually she succumbed to it. I slept in her room, I accompanied her on that last journey. And it kind of cemented my ideas that I would be a doctor and try and help people."

Diamond can't help but sound earnest a lot of the time - for all the "Jewish Elvis" glitz of his performances, it's just the way he is offstage - but even he admits that this idea was absurd. "But it was the way I felt then," he adds.

Once at college, Diamond found his attention wandering. He was fencing competitively with the varsity team and doing his biology homework on the subway to and from classes. And he was writing more and more songs. He bailed on college and began writing on Tin Pan Alley. It was the early 1960s and, pre-Beatles, the production-line style of songwriting was in its heyday. The likes of Neil Sedaka, Goffin and King, Lieber and Stoller were enjoying big success writing hits for popular groups of the day. But Diamond couldn't get a toe-hold, or a hit. He was repeatedly let go by music publishers.

"They were right to fire me - I thought the songs I was writing were uniformly terrible."

But he worked doggedly at his craft. He rented a storage room from a printer in the heart of Tin Pan Alley, above famous jazz venue Birdland. "I had a little annex room which had its own entrance into the hallway. Tiny room. I bought an old upright piano from a warehouse for 25 or 30 dollars. Had a payphone put in. Built a desk out of plywood. And I worked. I wrote out of there for a year. It was very convenient - just take the elevator down and you're on the streets, go and knock on doors. It was all fun. I wasn't on any kind of desperate search for success. I was basically just feeling my way around and trying to find a place for myself in the world of music."

Eventually he had his first minor successes, with Solitary Man and Cherry, Cherry. Don Kirschner, a music producer known as The Man With The Golden Ear, loved the latter song. He was looking for tunes for a new group he'd put together. He called Diamond: "Hey, you got anything for The Monkees?" Diamond played him some of the songs he'd recorded for his debut album.

"And Don said, Well, I'll take this one and this one and this one ' That was fine with me. Because I wasn't earning any money on my records. And The Monkees were having huge hits. And I needed the money. I had a baby, and my baby had to be fed. So I was very happy about that."

The Monkees' version of the Neil Diamond composition I'm A Believer was an instant smash. In America it was the biggest-selling single of 1967. Aged 26, after nine years of struggle, Diamond achieved lift-off in his musical career. "I had a tiger by the tail and I wasn't gonna let go," he says now. "I just kept on at it. And that became my life, writing for my records, and doing shows, for whoever would hire me."

The corridor leading from his front office to the rehearsal space is adorned with all of his album sleeves. On the cover of his debut, 1966's The Feel Of Neil Diamond, he sports a 1950s rock'n'roll quiff. "Yeah, that was pretty exciting," he says drily. On many of the others - including those released in the 1980s - he has elaborate 1970s hair. "I know, those were the days," he chuckles, although it has to be said that, as he contemplates an eight-month-long world tour, Diamond seems in good shape for a man nearing his 70s: decent head of hair, slim and spry.

We talk about the new album. Home Before Dark is a beautiful record, raw and unadorned. There are no hen-party anthems like Cracklin' Rosie or Song Sung Blue. But there are simple wee hours laments, and honest reflections from an inward-looking man contemplating a long life. One of the stand-outs is Another Day (That Time Forgot), an epic ballad on which Diamond duets with Natalie Maines of Dixie Chicks. His caramel baritone and her bell-clear country tones work together magically. It's his most high-profile vocal collaboration since he and Barbara Streisand united for the mega-selling You Don't Bring Me Flowers in 1978.

"When I wrote the song I heard a woman singing with me. Rick immediately said, Natalie Maines is a great singer, you should go to her first and see if she's interested.' I knew she was good but I didn't realise how good until we got into the studio. And she's more than just a pretty voice. She's a very sensitive and mature woman and she's open to opening herself up - which is very hard to find. So I was lucky to have her on this record."

He leads me into the studio, where his 14-strong group of musicians and backing vocalists are hard at work. They have a lot ahead of them. This Wednesday he comes to the UK to record a concert for Radio 2 (to be broadcast on Saturday). As befits the intimacy of Home Before Dark, this stadium-sized superstar will perform in the hushed environs of London's BBC Radio Theatre. For his next UK appointment after that he'll have to amp things up some: he's at Hampden, which is sure to be one long singalonga-Neil spectacular, powered by a stadium's worth of hot flushes from the female portion of the audience.

And then, Glastonbury - Diamond's booking to appear at the festival offering proof that, in this remarkable autumn of his career, he is still breaking boundaries and reaching new audiences.

"Working with Rick has certainly drawn a lot of attention to my music," he acknowledges, "which is important. You want people to listen to what you're doing and hear it. Rick knows how to stay out of the way, which is what a good producer does."

After 40-odd years and 120 million albums, what left is there for Neil Diamond to achieve? A lot, it seems. But for this stoutly contrary, proudly uncool, rebelliously middle-of-the-road songwriting genius, those goals are, typically, nothing to do with what we think of as "rock'n'roll".

"I'm a doting grandfather," Neil Diamond says. His oldest is 13, the youngest 14 months. "I adore them. Grandkids are the best because you can take care of them if you want to but you're not responsible for them 24/7." And again that lugubrious demeanour lets out a twinkle.

"And I'd like them to remember me," he continues. "That's one thing. I'd like them to remember me," he repeats, suddenly serious. "I don't wanna leave this planet until they're old enough to know me a little bit. I wanna be around so the baby Lily knows me."

He doesn't sound mordant, but sincere and defiant. He sounds like he does on Home Before Dark. It's another great, great Neil Diamond moment.

Home Before Dark is released by Columbia this week. Neil Diamond plays Hampden Park on June 5