There was a time, Shelley Klein remembers, when her parents used to stage catwalk shows in their Borders home. "All the fashion journalists from London would come up," she recalls as we sit in the house in question more than 40 years later. "They'd fly up, be brought down by bus here. We'd have all the models going around the house, showing the new collection."

She looks around the house – a clean-lined, beautiful, modernist vision called High Sunderland, designed by the architect Peter Womersley – recalling old times. "It's strange," she continues. "I can remember when I was very little going to Paris fashion shows. And I can remember a model giving me a beautiful pink feather boa. But that didn't seem odd to me."

And why would it? Alongside her brother and sister, Shelley Klein had grown up surrounded by textiles, by fabric, by colour, and by models ghosting around her living room.

We are sitting at a table in a house full of memories. She lives here alone now with her dog after her father's death last year. His paintings and fabrics are still on display. The house and everything in it is a testament to her father's taste.

Klein, now 52, is the daughter of Bernat Klein, a Serbian-born textile designer who transformed British textiles between the 1950s and 1990s and was lauded by everyone from Chanel to Marks & Spencer and worn by everyone from Jean Shrimpton to Princess Margaret. At his height he employed 600 people in his mill in the Borders and was given a CBE in 1973, one of numerous awards for his work. The Royal Incorporation of Architects Scotland even made Klein an honorary member because of his design skills back in 1980. When he died at the age of 91, RIAS secretary Neil Baxter pointed out that "for a generation of Scottish women, owning a Bernat Klein creation was an aspiration".

But that was then. His daughter now wonders if enough people know of her father to be able to even talk about his legacy. "Textile designers aren't well known." Perhaps. But here is one who deserves to be. This is Bernat Klein's story, as told through his daughter's eyes.

The Herald: The Bernat Klein studio designed by Peter Womsersley now derelictThe Bernat Klein studio designed by Peter Womsersley now derelict

Bernat Klein was born on November 6, 1922 in Senta, in what was then part of Yugoslavia. His parents Zorina and Leopold were orthodox Jews who ran their own wholesale textile business.

The outlines and textures of that childhood are inevitably a little faded to his daughter at this distance. "I think like most children you don't ask the questions when they're here and I'm kicking myself that I didn't. But I think he had quite an idyllic childhood there. He adored his mother.

"His earliest memories were the smell of the fabric, feeling the fabric, watching his parents work."

Klein grew up to be a liberal atheist but his background was strictly religious. At the age of 13 he was sent to a Yeshiva, a Jewish Orthodox school in Czechoslovakia, and in 1936 he travelled to Jerusalem to study with a rabbi.

Studying, he once said, amounted to the Old Testament, the Talmud and the commentaries. "Much time was spent in prayer and contemplation. Not having set foot in an art gallery, museum, concert hall or theatre, I was, at 17, almost totally unaware of the arts."

That changed in 1940 when he started attending the Bezalel School of Art and Craft in Jerusalem, where he learned to weave. By then the Second World War had started and the world had become a more dangerous place for the Jewish people. The temptation might be to believe that his parents sent him away from home because they could see what was coming. But Shelley doesn't think so.

"He lost all contact with them in 1941," she says. "Now I'm wondering why, given that Yugoslavia was invaded quite late on, why they hadn't fled. My surviving relatives who I've asked this question to said, 'Well, there was little communication. We didn't really know what was happening.'"

In 1944 Germany invaded Yugoslavia and Klein's family were sent to Auschwitz. "My grandfather survived but my grandmother died there. And various other relations didn't come through.

"I still have a cousin of my father's in America who was there. She was a little girl in Auschwitz under Mengele and was experimented on. She's 85 now."

Did that tragic history leave a shadow on their son? "You know, he was a very forward-thinking man. He was so close to his mother as a child. I think that did affect him terribly, but he was very determined. He had inner resources that, sadly, I don't think he passed on to any of his children."

What's amazing, I suggest, is that his work would prove the opposite of his own dark, doomy back story. His work was all about colour and life.

"When I trace his journey through life I just think he was extraordinary," Shelley agrees. "But I think you find a lot of refugees in that situation are extraordinary individuals who come a long way."

The Herald: Shelley Klein in High SunderlandShelley Klein in High Sunderland

Where did his love of colour come from? "I think it was probably there in him all the time. He recognised that his mother always dressed according to her eye colour so she always dressed in browns and greens. She had a great sense of style. He obviously picked up something there. Then, when he went to art college, he was being hit from all sides by new experiences. It very much became his raison d'etre."

After art college Klein lived in Egypt and had a short spell "spying" for the British. Well, spying might not be quite accurate. "His career with the British Intelligence service was very short-lived," his daughter says, laughing, "because he was not good at it. He was set to listen to radio broadcasts and translate them because he spoke five languages. He was told by his boss that if he thought anything was of interest he was to note it down. And he didn't think anything was of interest. And so he was asked to depart quite swiftly."

In 1945 he sailed to England to study textile technology at Leeds University. There he met Margaret Soper, whom he married in 1951. Two years earlier he had moved to Edinburgh to work with Munrospun, where he designed fabrics for women's coats and skirts before being transferred to Galashiels. It wasn't what either he or Margaret had expected or desired. That soon changed. "They actually both came to love the landscape around here and the people."

In 1952 he set up his own company, Colourcraft, and began to make lambswool scarves for Woolworths, British Home Stores and Marks & Spencer. Business took off and he commissioned Womersley to build a home for the family, which was completed in 1957. Klein chose all the fabrics and furniture. "His whole life was about good design in practice as well as theory. If anything didn't look right, out it would go."

Six years later Shelley was born in the same month Coco Chanel featured Klein's mohair tweed in her spring 1963 collection.

By now Klein was a minority shareholder in a company renamed Bernat Klein Ltd and his tweeds and fabrics were being used by the likes of Pierre Cardin, Christian Dior, Nina Ricci and Hardy Amies, among others. A year later his autumn 1964 couture fabrics included a velvet tweed that was lauded as "the fabric of the season".

His textiles offered a bold splash of colour amid the post-war drabness. "I think the man at Woolworths described his scarves as 'patisserie'," Shelley recalls. "They were that beautiful whirl of colour. It really took off."

As in work, so in life. "When we were little he'd buy us a lot of clothing from Scandinavia because they have lovely, bright, bold children's clothing which he loved. When all my friends were in jeans and navy jumpers I looked like a harlequin. And I craved tatty blue jeans and a jumper. When I look back we look fantastic.

"We went through all our different phases of fashion, much to his horror. I had a particular phase of Laura Ashley and, God, I think if he could have thrown me out of the house he would have."

What kind of father was he? "I would say quietly authoritative. He was a very gentle man with us but he definitely had a line over which we shouldn't step. We had huge freedom. He was very liberal. Both my parents were and they were a nice combination. He had the fiery Serbian temperament and my mother was very quietly English and very gentle.

"In his work I think he knew what he wanted and what he wanted he got. That was his work persona. But in the family he was much more giving."

Childhood, Shelley recalls, was spent in the woods around the house or playing around her father's feet as he painted in the house. Work and play were intertwined. "We'd go on lovely walks. He found a piece of lichen once and he painted it and that became a fabric."

Although he had a falling out with his backers in 1966 and left the company that bore his name, he quickly set up a new studio and the success continued into the 1970s. Womersley completed the Bernat Klein Studio near High Sunderland in 1972, a bold glass box of a building.

"My sister had either her 18th or 21st birthday there. I remember people stopping on the road thinking it was a restaurant because it was so lovely. He had all his orange and red paintings around the central core so at night when it was lit up it really glowed. It's an extraordinary building." Unfortunately it is now empty and on the Buildings at Risk register.

The 1980s began to see a downturn in Klein's fortunes as the recession bit. He retired in 1992 to concentrate on his painting.

The paintings and a series of tapestries Klein commissioned based on his paintings back in the 1970s are now the heart of a new exhibition at the Dovecot Gallery in Edinburgh. It's a chance to reassess Klein's artistic vision. His fabrics and paintings sing with colour. The warp and weft of his vision extends to everything he turned his hand to.

Margaret sadly died in 2008 and Shelley moved home a year later to help out. It sounds, I suggest, like her dad had a good retirement. "That's what I was hoping to give him by coming back. It's not that he wasn't capable of looking after himself. He had various illnesses but he was in very good shape. He did exercises every day – sit-ups and press-ups."

And he would talk about art and fashion, the things he loved most. "I'd have friends to stay and he'd ask, 'Which artists do you like?' If you said Andy Warhol he'd probably raise his eyebrow. He was very fond of Seurat and Cezanne and Vuillard. We used to have a little Vuillard but sadly not now."

As for the contemporary fashion scene, he retained an interest, Shelley recalls. He admired Alexander McQueen for the cut of his clothes. Galliano less so. The designer's drunken anti-Semitic outburst didn't help, of course.

"I think he found the whole fashion industry these days quite obscene. How much everything costs, the importance of celebrities. It's a different ball game."

Life inevitably became more restricted for him. But not restrictive. "He missed my mother terribly. That was the hard thing. But, again, he made a life after her death. I hope it was a good retirement."

Shelley points across the open-plan room to a painting. "The one on the easel is the last painting he did and he completed that about a week before he died."

As well as running her greeting cards business, Shelley is now writing a book about High Sunderland – the pleasures and pitfalls of living in a modernist house ("no insulation, heating bills you would not want to swap with me and a flat roof in Scotland".) Inevitably, though, it is also her father's story. He is written into the stone of the place. And into his daughter's heart.

The afternoon ticks on. I leave Shelley Klein in a home full of memories. None of them is in black and white.

Bernat Klein: A Life in Colour opens at the Dovecot Studios on July 21 as part of the Edinburgh Art Festival and runs until September 26.