First Minister Alex Salmond praised the anti-apartheid icon as an "inspiration to countless millions".
Former prime minister Gordon Brown said the 95-year-old had been "the greatest leader of our generation".
Flags are flying at half-mast across Scotland, including at the Scottish Parliament and in Glasgow, the first city in the world to grant him Freedom of the City.
The award was bestowed upon Mr Mandela in 1981 while he was imprisoned on Robben Island.
The city continued to mark its support for the campaign to free him in 1986 when it renamed St George's Place as Nelson Mandela Place in his honour.
Thousands gathered in George Square when he finally visited the city to receive the award in October 1993, the year before he became president.
Mr Mandela thanked the citizens of Glasgow and said the award was a "tremendous" personal honour.
"While we were physically denied our freedom in the country of our birth, a city 6,000 miles away, and as renowned as Glasgow, refused to accept the legitimacy of the apartheid system and declared us to be free," he said in a speech in the city.
Glasgow's Lord Provost, Sadie Docherty, described Mr Mandela as a political and moral icon.
"Nelson Mandela dedicated his life to bringing freedom, justice and equality to the people of South Africa," she said.
"His beliefs cost him years of his own freedom but his vision for peace and democracy prevailed. His legacy will live on and inspire generations to come.
"Glasgow was proud to be the first city in the world to honour him with a Freedom of the City award and he will be sadly missed by a city which had the greatest of respect for him."
A book of condolence is available to sign in the city chambers foyer.
People gathered to celebrate his life at Nelson Mandela Place, organised by ACTSA Scotland, successor organisation to the Scottish Anti-Apartheid Movement.
The First Minister said: ''With the passing of Nelson Mandela, the world has lost a towering statesman and the outstanding political leader of his generation.
''Mr Mandela's integrity, humanity and compassion were an inspiration to countless millions around the globe and his influence transcended ideology, race and creed.
''He was also someone who had a long-standing commitment to and friendship with Scotland, and I had the privilege of meeting him once. Those links with Scotland were underlined by his being granted the Freedom of the City of Glasgow in 1981 when he was still imprisoned - the first city in the world to do so."
Holyrood's Presiding Officer, Tricia Marwick MSP, has written to the South African high commissioner to offer condolences on behalf of the Parliament.
"Nelson Mandela was an inspirational figure, not just in South Africa but throughout the world and particularly here in Scotland," she said.
"His affection for Scotland was returned when he visited Glasgow in 1993, where he recognised that the city had been the first place in the world to grant him Freedom of the City while he was still imprisoned on Robben island.
"On behalf of all members of the Scottish Parliament, I extend our sincere sympathies to Mr Mandela's family and to the people of South Africa."
Mr Brown said: "Nelson Mandela was the greatest leader of our generation. A leader of magnanimity, fortitude, unshakeable optimism and, most of all, the most courageous man I ever met.
"True courage requires not only strength of will but strength of belief. What motivated Nelson Mandela and drove him to risk his life for freedom was a burning passion that irrespective of colour, race and background, all people are created equal, and his list of historic achievements starts with a multiracial South Africa."
Scottish Labour leader Johann Lamont said: "He fought injustice when there seemed to be no chance of victory. But he fought, he endured and he won.
"Nelson Mandela was the best of Africa. He was the best of humanity. He was the best of us all."
Glasgow's Archbishop Philip Tartaglia, president of the Catholic Bishops' Conference of Scotland, said: "I will remember Nelson Mandela not only for his courage and his ideals; rather I will remember him for the great example he gave of the power of forgiveness. And from his forgiveness great hope grew."
For the Church of Scotland, the Right Rev Lorna Hood, the Kirk's Moderator, said: "Nelson Mandela was a towering figure of the 20th century whose strength, courage and determination are only matched by his grace and ability to forgive.
"He will forever be remembered not only for the end of apartheid in South Africa but the manner in which the change was accomplished.
"Emerging from prison after 27 years in Robben Island, without bitterness or a call for revenge, he led by example believing that the only hope for his country was the reconciliation of all people regardless of their colour or creed."
Lord Provost of Aberdeen George Adam said: "Alongside many other cities, Aberdeen awarded Mr Mandela our highest honour, the Freedom of the City, a token of our respect and admiration for a man whose campaign for freedom and democracy had a global impact and whose legacy will inspire generations to come."
Unison said its predecessor, Nalgo, gave him honorary life membership of the trade union in 1984.
Scottish secretary Mike Kirby said: "Whilst we are sad today at the passing of one of the great men of our time, we celebrate his life of achievement and we take pride in the fact that this union has never wavered in our support for freedom and justice in South Africa."
Scottish Conservative leader Ruth Davidson said Mr Mandela was "a giant of our age".
"He healed his nation and, in doing so, inspired millions," she said.
"He showed the world that reconciliation could be a more powerful force than retribution."
Mr Mandela had been patron of the Scottish Refugee Council since 1995.
Judith Robertson, chair of the board of directors, said the charity is "deeply grateful" for his support and will continue to take "hope, inspiration and dignity" from his life.
Professor Alan Miller, chairman of the Scottish Human Rights Commission, said Mr Mandela transcended race, class and nationality to "epitomise the human spirit" and "bring out the best in all of us".
EIS general secretary Larry Flanagan was a spokesman for Glasgow on equality and anti-apartheid during his time as a city councillor.
"Nelson Mandela's lifelong commitment to freedom, equality and social justice is an inspiration to billions of people around the world, and to millions of people here in Scotland," he said.
"His links with and affection for Scotland were well known and many teachers and lecturers, students and pupils were privileged to hear him speak during his visit to Glasgow in 1993.
"It is impossible to overstate the impact that he had on all who met him or heard him speak, and the legacy that he leaves behind is one of continuing hope and belief in the inherent goodness of humanity."
The South African national flag is flying at half-mast alongside the Saltire outside Scottish Government headquarters at St Andrew's House in Edinburgh today.
The renaming of Glasgow's St George's Place as Nelson Mandela Place was particularly significant because the South African consulate was located in the street.
Mr Mandela was elected honorary member of the students' association at the universities of Glasgow and Strathclyde in 1984, the same year he was awarded the Freedom of the City of Aberdeen.
In 1990 Glasgow Caledonian University was the first to award him an honorary doctorate on his release from prison.
He was granted Freedom of the City of Edinburgh in 1997.
Dr Michael Kelly was Lord Provost of Glasgow in 1981 when the city became the first to honour Mandela.
He said the mood at the time was not as it is now and he had to convince people to support the award.
"I'm very proud to have played a small part in Mandela's fight against apartheid," Dr Kelly said.
"In the 1980s the anti-apartheid movement approached Glasgow Council and we looked at Mandela's case and history and the injustice in South Africa and we decided that the best way to bring his case to the attention of the public was to make him a Freeman of the city, and we did that at a time when his cause was not that well known or supported in Scotland or the UK so we had to convince people that this was a man and a cause worth supporting.
"It's great that Glasgow was on the right side of history because we had to make a decision on it and there were a lot of forces telling us to just leave it alone, but we made the right decision and it was great that other people followed through, for example I was invited to address the United Nations which had organised a day of world mayors supporting Mandela, so from small beginnings in Glasgow it became a big part of the international movement to get him free and to get a proper democratic regime established in South Africa."
When Mandela came to Glasgow in 1993 he told Dr Kelly that hearing of the support he had in Scotland, and across the world, gave him great encouragement.
"I asked him if he knew about the protests and demonstrations and awarding of the Freedom in Glasgow and he said yes. He told me there was a very sophisticated grapevine (in Robben Island), that the authorities did not want him to know what was happening in the outside world because they were trying to convince him that his case had been forgotten, but knowing he was made a Freeman of Glasgow and things like that kept him going because he realised the fight was worldwide and there were plenty of people supporting him."
Dr Kelly is also a former director of Celtic and gave Mandela a gift when he arrived in Glasgow.
"After the Freedom ceremony I handed him a Celtic scarf and he smiled. He assured us that he would keep up the connections with Glasgow and I think that part of his legacy is for Glasgow to ensure that we are at the forefront of combating any form of racism.
"I think he will be remembered as a man whose life was fulfilled. He set out with a political ambition to change society and despite extreme difficulties in his life he was able to die knowing that he had achieved that. It's a fulfilled life."