Mr Jardine, who played for the club 451 times and made 184 appearances latterly for Hearts, had been diagnosed with the disease in his liver. He passed away peacefully surrounded by his family yesterday.
Rangers said the football world had "lost a man of dignity, class and honour".
McCoist said: "We are all devastated with the news he has passed away, we have lost a great man.
"I had the privilege of watching Sandy playing for Rangers when I was a young boy, I had enjoyed the pleasure of working with him closely since I returned to the club in 2007 and he was a truly remarkable human being.
"His achievements both on and off the pitch are second to none and I was honoured to regard him as a friend.
"He was respected not only by Rangers fans but also the wider football community and he is a huge loss to the game.
"We will never see his like again in the modern era."
Former Manchester United manager Sir Alex Ferguson, whose Aberdeen side beat Hearts in the 1986 Scottish Cup final when Mr Jardine was by then playing for the Edinburgh club, sent condolences from him and his wife Cathy.
He said: "This is some of the worst news we have heard.
"Sandy was a noble and courageous man. The respect he is held in at Rangers is immense.
"He was one of the greatest players ever to wear the jersey. To Shona and family, we express our sympathy and sadness."
Mr Jardine first played for Rangers in 1967, aged 18, before going on to make almost 800 appearances for the club.
The defender won three League Championships with the Ibrox side, five Scottish Cups, five League Cups and the European Cup Winners' Cup in 1972.
He also featured in two World Cups, winning 38 caps for Scotland.
McCoist recalled how Mr Jardine had told him how it had been his life's ambition to play for Rangers.
The former player added: "He recently told me he was proud to be a Ranger and wanted to be remembered forever as a Ranger.
"Well Sandy, you will go down in history as one of the greatest of all time and we will miss you terribly. It is a very sad day for everyone associated with Rangers."
Mr Jardine, who leaves behind wife Shona, children Steven and Nicola and grandchildren, was diagnosed 18 months ago.
During his successful career with Rangers and Hearts he was twice named Scottish player of the year.
Mr Jardine first won the accolade with Rangers in 1975 and, at the age of 37, with Hearts in 1986, after the club had narrowly lost out on the league title to Celtic.
In a statement, the Tynecastle club said: "Heart of Midlothian FC is deeply saddened at news of the death of club legend Sandy Jardine. Our condolences go to his family at this time."
The Scottish Football Association also offered its condolences.
Rangers player Steven Smith said on Twitter: "Strong, proud, loyal and dignified.
"He is everything a Ranger should be. Rest in peace Sandy Jardine."
Mr Jardine was instrumental in the creation of the the club's Hall of Fame.
He was the driving force behind the Rangers Fans' Fighting Fund which was launched after Rangers went into administration two years ago while under Craig Whyte's reign.
He led a march of thousands of Rangers fans to Hampden Park to protest at the sanctions imposed on the Club that threatened its existence at that time.
A spokesman for Rangers fans group, the Union of Fans, added: "Through some of the club's darkest moments, he stood side by side with the fans."
Celtic chief executive Peter Lawwell said it was "absolutely devastating news".
He added: "Our thoughts and prayers are very much with Sandy's wife Shona, his children Steven and Nicola, and the wider family.
"He was a very fine man and it was a privilege to know him."