His uncle - his mother's younger brother - was killed in the First World War and the family became Quakers soon after. That meant that Rorem himself was exempt from military service on religious grounds; now 90, he remembers that "no, it wasn't especially easy" to be a prominent conscientious objector in the US during the 1960s.
Rorem has lived an extraordinary life. We know about it in extraordinary detail, too, thanks to the notoriously candid diaries he wrote during his Paris years in the 1950s (and later when he returned to the US). As a beautiful and talented young flâneur he quickly found himself moving in the social spheres of Poulenc, Dali, Bowles, Cocteau and Cage. The diary doesn't hold back in its high-society name dropping, gossip, scandal and bawdy nocturnal exploits. But it also offers a remarkably frank insight into the creative process of a fledging artist.
Rorem went on to become one of the most prolific American composers of the 20th century. He has written 10 operas, three symphonies, multiple concertos, dozens of chamber works, ballet scores and theatre music. He has published 16 books of prose, including diaries and volumes of music criticism. But it's his songs - more than 500 of them - that form the backbone of his musical output and the most concise summation of his literary and musical talents. For Rorem, words and music best go together.
Thematically they range all over the place, from love to loss to longing and other art-song preoccupations. But it's his war-themed songs, or rather his pacifist songs, that bring his music to Scotland next month. To mark the 100th anniversary of the start of the First World War, the Hebrides Ensemble have built a programme around the themes of tragedy, demoralisation, defiance and hope. They are joined by baritone Marcus Farnsworth - the grippingly visceral soloist in their recent performances of Sir Peter Maxwell Davies's Eight Songs for a Mad King - who sings George Butterworth's heartrending setting of Housman's A Shropshire Lad and a new work by Stuart Macrae called Parable, after Wilfred Owen's The Parable of the Old Men.
Book-ending the programme are two extracts from Rorem's 1969 song cycle War Scenes. For a composer whose work is almost always tonal and whose vocal lines are often folksy, War Scenes is startlingly acerbic, angular, rugged music. The motivation behind it is unequivocal. Rorem took his texts from Walt Whitman's autobiographical Specimen Days, a reflection of Whitman's time as a volunteer field nurse during the American civil war. The short dedication at the front of the score reads as follows: To those who died in Vietnam, both sides, during the composition: 20-30 June 1969.
When I ask Rorem about that dedication now, he says that he meant exactly that, nothing more or less. "This music was intended to commemorate those who died in Vietnam," he says. "Those on both sides. Sure, it was a provocative thing to write at the time. I knew it would be. But I'm against all war under any circumstances. How could I have written anything different?"
Rorem says that of the vast number of poets whose words he has set to music (from Shakespeare, Tennyson and Yeats to Frost, Cummings, Sylvia Plath and Gertrude Stein; he almost never uses his own texts), Walt Witmann stands out as "very singable - wonderfully singable. His language is special, very open somehow." He asks me whether Farnsworth has good diction when he sings. I say that he does. "Good," he replies, "because the words must come before music. They always must. Clear words are more important than a beautiful voice."
This assertion will come as no surprise to performers of Rorem's music or readers of his music criticism: he has long been of the opinion that music has no inherent meaning. Instead he broadly agrees with Stravinsky's doctrine of 1936: that music is, by its nature, essentially powerless to express anything, whether a feeling, an attitude of mind, a psychological mood, a phenomenon of nature, etc. "Yes, that's about right," says Rorem. "A composer can tell you that his symphony represents a zoo in Turkey if he wants to, and the audience will believe him. But it's only ever going to be his own interpretation, nothing more."
Vocal music, though, is different. "In vocal music the text carries everything," he says. "The text can carry a world of meaning."
And so the opportunity to blend music's emotive triggers with the world of meaning possible from words has drawn Rorem back to songs throughout his career. His earliest teenage musical sketches were impressionist songs with titles such as The Golden Nightingale ("very French, don't you think?", he laughs). He's happy to admit that "whatever talents I might or might not have, I think it's fair to say I've got pretty good taste in literature. I know how to choose a nice text."
Over the phone I can hear him take a sharp breath in. "Actually I'm a singer too," he adds, a little conspiratorially. "It's just that nobody ever wants to hear me."
There's a bemused frankness and concision to Rorem's conversation. He once said of his Paris Diary that he was "merely too lazy to pretend to be something I'm not". Whether or not the root is laziness, I get the feeling that he has neither the need nor the patience to dress up the truth. When I ask how, as both a writer and composer, he chooses which medium to turn to on any given day, he answers without hesitation. "Money. That's easy. I never work without a commission." Is he working on anything new at the moment? The response is similarly unguarded. "Nah, nothing new. To be honest, I think I've already written almost everything I have to write in both prose and music."
His tone isn't wistful or pitying, just matter of fact. Later, when pushed, he admits that he "wouldn't mind" writing another opera. "I'm too old to write a full-length one; maybe a little one-acter would do." For now, though, he seems justifiably content with the legacy he has already created.
The Hebrides Ensemble performs extracts from Ned Rorem's War Scenes at the University of Glasgow Memorial Chapel on February 10; Queen's Hall, Edinburgh, on February 11; Perth Concert Hall on February 12 and The Sage, Gateshead, on February 13.