But unlike other camps on the island, this one had an art studio as many of the inmates had artists classed as "degenerates" by the Nazis in their homeland.
Now a major exhibition is under way at Tate Britain of the camp's most celebrated artist, Kurt Schwitters, who was to have a profound impact on the British art scene.
And the Scottish son of the camp commandant, Hubert Daniel, has revealed how it was his father's determination to let the artists work that allowed them to flourish .
Landscape architect Peter Daniel says his father did all he could to nurture the talents of those in his charge.
Peter Daniel, now 88, travelled from his home in the Scottish Borders to the Tate to see the Schwitters exhibition, which owes its existence, in part, to the kindness his father showed to those captive artists.
Daniel said: "My father didn't have to do what he did, but he did, fortunately. Perhaps because he survived the first world war and maybe he had a certain sympathy for the underdog.
"He found them materials, gave them space, a studio, allowing them to meet and talk and generally get on together. They were all academics and artists fleeing the Nazis.
"The studio was rough-and-ready, made out of a half-constructed house. They were able to convert it into a studio.
"One of the things my father did when he got to the camp was get the dieticians together and produce a menu of what the internees should be fed on – porridge and herring.
"We know Schwitters didn't like the porridge but he collected it from all the other prisoners and made sculpture from it, a huge thing.
"It was so big the floor of the boarding house collapsed into the floor below.
"He was an abstract artist, he collected everything."
Daniel said a number of the artists collaborated on a book of sketches, and gave his father the album in appreciation of his efforts.
Schwitters's impact on British art in the immediate post-war era was profound, resonating with artists from the pop generation of Richard Hamilton and Eduardo Paolozzi to artists working today.
On release in 1941 he became involved with the London art scene and his work began to engage with British society and culture, shaped by the use of a new source of materials for his collages and poems.
In 1945, Schwitters moved to the Lake District and his work focused on the natural world.
His last great sculpture and installation, the Merz Barn, was created there.
He died in Kendal in 1948, the day after he was notified of his successful application for British citizenship.
For Peter Daniel, seeing Schwitters's work at the Tate, including the sketch book gifted to his father, was an "amazing experience".
He added: "I saw the sketch book. It's amazing. It's all so different. It ranged from ordinary portraits to abstracts.
"I'm just thankful my father's been recognised after all these years.
"He was chucked out of the army after the war and died a pauper. He had no pension, nothing."