While David Cameron was issuing assurances yesterday that he will not "lurch to the right" after his party was left trailing Ukip in the Eastleigh by-election, his tough-talking Justice Secretary was letting it be known that a future Conservative government would scrap the Human Rights Act.

Chris Grayling's statement, clearly designed to placate right-wingers, goes further than any other minister since the Coalition was formed. He does not specify which human rights the Tories would abandon but says he wants "a dramatically curtailed role for the European Court of Human Rights in the UK". Meanwhile, Home Secretary Theresa May is reported to be working on plans for Britain to withdraw from the European Convention on Human Rights (ECHR) if the Conservatives win an overall majority in 2015.

Those who believe the UK should represent the gold standard in protecting human rights must take this threat seriously. The paradox is that while the Human Rights Act is often depicted as being a European imposition, the UN Declaration on Human Rights, on which it is based, is basically a British export and one we should be proud of. It was framed as Europe emerged from the horrors of the Second World War and was forged from a determination not to repeat such atrocities. Britain was one of the first signatories in 1948.

Yet from the moment the Labour Government enshrined human rights in domestic law in 1997, much wilful misunderstanding has gone into the reporting of this issue. Certain sections of the media have used it as a cipher for knee-jerk anti-Europeanism and a vehicle for demonising asylum seekers.

Yards of column inches have been devoted to often ill-informed stories about prisoners and terror suspects using the legislation. For instance, murderer Denis Nilsen did not, as was claimed, use the act to access pornography in his cell. The case was thrown out. The legislation has been used to prevent the deportation of terror suspects to countries that employ torture, but the UK needs to be consistent. The Government cannot oppose torture but then allow certain people to be tortured. Human rights are by their nature universal or they are worthless.

It is less well known that the Human Rights Act has upheld the rights of kinship carers. It has laid a duty of care on the Army after a British soldier in Iraq died not from enemy fire but heatstroke. It has given deaf patients the right to an interpreter during operations under local anaesthetic. It has obliged local authorities to consider the emotional and physical health of residents before closing care homes and enabled a couple married for 65 years to be cared for together.

It is up to supporters of human rights to challenge the mythology that has grown up around this legislation. The UK Government very rarely loses a case in Strasbourg. By contrast the ECHR has had a huge impact in countries such as Turkey and Russia, with poor human rights records. What sort of a message would the UK be sending out to such countries by withdrawing from the convention? And what moral authority would our Government have in calling for an end to rights abuses elsewhere? As Professor Alan Miller, Scotland's Human Rights Commissioner, says in The Herald today, such an unprecedented retreat from accountability risks "undermining our whole human rights system".