Imran Shahid, 37, wanted a £6000 payout from the Scottish Government and a declaration that his treatment was unlawful after he spent almost five years away from the mainstream prison population.
Shahid was ordered to serve a minimum jail term of 25 years under a life sentence after he was convicted in 2006 of the racially motivated murder of 15-year-old Kriss Donald in Glasgow.
He raised a claim maintaining the decisions to keep him virtually continuously segregated between October 2005 and August 2010 violated his rights under articles of the European Convention on Human Rights protecting against torture and cruel punishment and the right to private life.
The bid was originally rejected in 2011 at the Court of Session in Edinburgh, but the case was appealed and three judges at the court yesterday refused the latest legal challenge.
Lord Drummond Young, who heard the appeal with Lord Menzies and Lord Wheatley, said: "The simple fact is that continuing threats to his personal safety were made. In those circumstances there was no alternative to segregation."
He said it was clear from evidence available to the prison authorities that "serious threats of harm" to Shahid had been made by other prisoners.
One report from 2006 said prisoners at Glenochil jail in Clackmannanshire had warned that if any of the schoolboy's murderers were housed there they would be murdered, and "there would be a queue of prisoners wanting to do it".
Lord Drummond Young said the appeal judges were "quite satisfied that adequate grounds existed for the continued segregation".
Shahid was jailed with others after Kriss Donald was abducted in Glasgow in March 2004 and subjected to a terrifying journey before being stabbed and set on fire.