But now the industry is toasting its most successful year for more than a decade after enjoying the attentions of celebrities such as Dr Who's Matt Smith, Kylie Minogue, Madonna and Gwyneth Paltrow.
As well as being worn by the famous and fashion conscious and carried in the form of fashion label Radley's bags, the fabric also provides the inspiration for an upmarket hotel in Glasgow and motorcycle jackets in the US.
The cloth that was most commonly associated with the gent's sports jacket, is also now a fashion feature from South Korea to Russia.
Yesterday, Harris Tweed Industry Forum meeting in Stornoway revealed 2012 has been the best year for production of the cloth in nearly 15 years.
Since 2009, the body has collaborated on a strategy to support the growth and development of the industry. In particular, a joint approach to skills and training has paid dividends.
Nearly 40 new Harris Tweed weavers have entered the workforce in the past four years, and an increasingly structured approach to training means the majority of weavers are now trained to SVQ Level 2.
Ian Angus Mackenzie, chief executive of Harris Tweed Hebrides, said: "We are delighted with the progress the industry is making – it has been a good year for the mills and the weavers.
"We are confident we currently have the right balance in terms of number of weavers, weaver capacity and market demand.
"We are looking to continue to grow the value and diversity of the international market for the cloth, and this incremental growth will be supported by existing weavers and new entrants due to natural turnover within the workforce."
This view was endorsed by The Carloway Mill Harris Tweed, where Radley + Co earlier this year secured its very own tweed which is being produced at the factory for sale in its stores, including in London's Kings Road and New York.
The mill's Bruce Armitage added: "It is important to continue to maintain a balance between the increased market demand and the ability of the industry to deliver to the market; including a stable weaver population."
Lorna Macaulay, chief executive of the Harris Tweed Authority, said: "I am pleased to confirm nearly 40 new weavers have entered the workforce over the past four years and are now making a significant contribution to production capacity, which has been met with an increasing appetite for the cloth at home and overseas."
Other successes for the industry, include the decision to include Harris Tweed materials in some of the suites at the luxury Blythswood Hotel in Blythswood Square.
Smith also sparked interest in the design from the younger generation when he donned a vintage Harris Tweed jacket when he took on the lead role in Dr Who two years ago.
However, it was reported a revised version of the costume worn in subsequent episodes was being imported from China.
The Industry Forum, which consists of representatives from the Harris Tweed Authority, the Weavers' Association, the Harris Tweed Mills, Comhairle nan Eilean Siar (Western Isles Council), and Highlands and Islands Enterprise, also discussed initial findings of a study into loom enhancement and modification, and agreed to prioritise progress in this area over the coming year.