The "Robben Island Bible" - which is actually a copy of the complete works of Shakespeare - belongs to a Sonny Venkatrathnam, who was in the jail at the same time as the former South African president.
It has gone on display at Glasgow's Mitchell Library to coincide with Mandela Day and the book will be on show there for the next three months.
Mr Venkatrathnam has loaned the book, which he kept in his cell after disguising the cover with Diwali cards and telling prison guards it was a bible to prevent it from being seized, to the city.
While in prison he passed it to 33 of his fellow inmates, asking them to sign a passage that meant something to them.
Mr Mandela, who died last year at the age of 95, underlined a passage from Julius Caesar and signed his name beside it on December 16 1977.
The passage he highlighted reads: "Cowards die many times before their deaths. The valiant never taste of death but once.
"Of all the wonders that I have heard, it seems to me the most strange that men should fear, seeing that death, a necessary end, will come when it will come."
Mr Mandela was granted the freedom of Glasgow in 1981 when still a prisoner in Robben Island. Five years later the city again honoured the anti-apartheid campaigner, renaming St George's Place after him.
After he was released from jail, Mr Mandela visited Glasgow in 1993 to thank the city for its support.
Karen Cunningham, head of Glasgow libraries, said: "Nelson Mandela had a great affection for Glasgow and we are extremely proud of our strong connection to such a remarkable man.
"He was arguably the greatest Commonwealth leader of all time and therefore it is fitting that we celebrate his life and achievements as we host the 2014 Commonwealth Games.
"The Robben Island Bible is of strong historical significance and we are grateful to Sonny Venkatrathnam for loaning us the book.
"I'm sure many visitors will come along to the Mitchell Library to see it in person."
Meanwhile, Mandela's granddaughter has thanked the people of Glasgow as she joined celebrations of the anti-apartheid icon's life.
Tukwini Mandela was the guest of honour at events held in the city to mark this year's Nelson Mandela International Day.
Hundreds turned out in Glasgow, the first city in the world to honour the former South African president with the freedom of the city in 1981, to celebrate Mandela's life and the values for which he stood.
His granddaughter attended a ribbon-tying ceremony in the Glasgow street that bears her grandfather's name, Nelson Mandela Place.
Guests at the ceremony tied ribbons in the colours of the South African flag to the railings of St George's Tron Church as the first Mandela Day since the former leader's death in December was marked.
Tukwini Mandela said that it was a bittersweet day because it would have been her grandfather's 96th birthday.
She said: "I'm here in Glasgow just to thank the people of Glasgow for all of their support.
"I know that Glasgow was one of the first cities that awarded my grandfather the keys to the city.
"It galvanised a lot of the European cities to pay attention to what was going on in South Africa.
"So, my family and I thought it would be great to come here and thank the people of Glasgow."