Scottish ministers are bracing themselves for hundreds of compensation claims after a landmark Supreme Court ruling that they have acted unlawfully.

Five law lords yesterday issued the first constitutional ruling of its kind in finding successive Labour and SNP governments had failed to protect the right of hundreds of mental health patients to appeal against the level of security imposed on them.

The UK's highest court for civil appeals condemned the failure since 2006 to pass the necessary laws that would allow patients in non-state hospitals to lodge an appeal against their conditions of detention.

It is the London-based court's latest judgment overruling Scottish legal decisions; actions which have been criticised by First Minister Alex Salmond in the past.

While the Mental Health Act allowed for such an appeal, a lack of action by ministers resulted in those detained being denied an appeal against their level of security, the court ruled.

It follows a 2006 test case brought by a man detained at Leverndale Hospital in Glasgow. Patient "RM" had been institutionalised for 35 years after a road accident left him with significant mental health difficulties.

The legal battle began when a tribunal found he was unable to lodge an appeal against his security level because the legislative statutory instruments to the Mental Health Act 2003 had not been passed.

His solicitor Frank Irvine argued his client had been detained in conditions of excessive security and that his quality of life, liberty and his prospects for release would be improved were he to be transferred to an open ward.

Lord Hope, Lady Hale, Lord Wilson, Lord Reed and Lord Carnwath unanimously upheld the patient's appeal against an earlier Court of Session ruling, finding: "The failure by the [Scottish] ministers to draft and lay regulations, and their continued failure to do so since [May 1, 2006] was and is unlawful."

Mr Irvine said: "The implications of the ruling are fairly significant and immediately there are several hundred people who should now have the right to appeal.

"The question of damages for patients who have been unable to appeal since 2006 and the suggestion of compensation following this ruling of unlawfulness is now a live one.

"This ruling is to be welcomed not just for our client but for the many individuals who have been denied a right of appeal enshrined in law and passed by the Scottish Parliament."

The court said the failure to make regulations which would implement a right of appeal for patients had "thwarted the intention of the Scottish Parliament" because "it is for Parliament, not the Executive – unless Parliament confers the necessary power upon it – to determine when an enactment comes into force".

It has never before been held in Scotland that ministers were bound to bring statutory provisions into effective force.

In May 2011, Nat Fraser was freed from prison after the court quashed his conviction for the murder of his missing wife Arlene in Elgin, but he was later found guilty after a retrial.

Mr Salmond later slapped down Lord Hope and criticised leading human rights lawyer Professor Tony Kelly in a magazine interview.

But Niall McCluskey, QC, said: "The key issue here is the rights of the individual, a mental health patient, who was unable to bring an appeal about the standard of his detention."

Human rights lawyer John Scott said: "The Mental Health Act was intended to allow our human rights legislation to be human rights proofed and to try to thwart it on still an unexplained basis flies in the face of that."

Billy Watson, chief executive at the Scottish Association for Mental Health, urged the Government to respond swiftly to the ruling. Public Health Minister Michael Matheson said the Government was considering the terms and implications of the judgment.