Charles McCann secured a ruling that the 2011 move at the State Hospital at Carstairs was unlawful and breached his human rights in a judicial review last year at the Court of Session, Edinburgh. But following an appeal by the hospital authorities three judges have now reversed the decision.
The Lord Justice Clerk, Lord Carloway, said the ban was "proportionate to the legitimate aim of promoting the health of those detained and those at work".
McCann, a schizophrenic, spent 18 years in the institution. He was later moved to a medium secure unit.