Something strikes me as not quite right about the idea that an unelected civil servant should appear on television rubbishing the policies being proposed by a minister in the government which it is his job to serve.

Yet the BBC does not seem to find it odd that Keir Starmer, the outgoing director of public prosecutions, should be invited on to Andrew Marr's sofa to do just that. It's not, however, surprising that Mr Starmer, whose background is in human rights law, should dislike the Justice Secretary's intention to scrap the Human Rights Act. One of the advantages of incorporating the European Convention on Human Rights into English law, from his point of view, is that the frequent need to interpret it puts power in the hands of people like him; judges, lawyers and unelected technocrats.

For those of us who believe that laws ought to be made by the people who are elected to do it, this is a bad thing. But for those who would like to scrap the human rights act (HRA), the most obvious question is the one which the late Lord Bingham put after listing some of the rights protected by the ­Convention and the Act: "Which of these rights, I ask, would we wish to discard?"

Since they include the right to life, the right not to be tortured or enslaved, the right to free elections, freedom of expression, assembly and religion, the right to a fair trial and several other similarly uncontroversial ambitions which, I take it, most people approve of, the answer is presumably none of them. Unless you're a fascist, a Stalinist or sympathise with the Taliban (and I have no reason to think Chris Grayling falls into any of those categories), these rubrics admirably describe the state of affairs we would like to be the case. They are what Sellar and Yeatman would have called A Good Thing.

Naturally enough, the Government doesn't actually want to ditch them, not least since the principles behind them were present in the laws of England and Scotland long before the ECHR was drafted (by a Scotsman) in 1950, its incorporation in 1998 or when it came into effect in 2000.

Its proposed solution is to include them in a British Bill of Rights. In part this is motivated by a desire to avoid embarrassing cases where it proves difficult to deport criminals, and judgments such as the one which ruled that convicted criminals were entitled to the vote.

Oddly enough, the stated purpose of incorporating the ECHR through the HRA was to reduce the need to appeal to the Strasbourg court, but in practice it has meant that the British courts need to take into account the compatibility of its rulings with ­legislation - can, indeed, strike down secondary legislation which does not comply with it (though for Acts of Parliament, judges and tribunals can only issue a declaration of incompatibility).

Though the HRA made it difficult to deport a pest such as Abu Qatada, and that was irritating and inconvenient, it's not much of an argument for ignoring the law itself. Safeguards against locking people up without trial because you don't like the look of them, or shoving them on a plane back to a country which they may have fled in genuine fear of persecution, seem perfectly reasonable. It's in their nature that they will have the annoying consequence of also protecting people who are a bad lot.

But the reason why I think both Mr Starmer and Mr Grayling are ­barking up the wrong tree is that the whole notion of rights is based on looking at the world through the wrong end of the telescope. I hasten to add that I do not mean what rights are designed to ensure, or describe; I have no hesitation in approving of life, liberty and all that jazz. It is merely that rights, like laws, are human constructions, expressed by means of language, another human construction. And like all expressions of moral ideas or intended effects, they can be ambiguous, they can contradict one another, and there is a limit to how well they correspond to the world.

The very language of rights is designed to suggest that they are immutable and eternal. Some people, such as Hillary Clinton addressing the UN in 1997, believe that: "They have been with us for ever, from civilisation's first light." In support of this notion, she cited the moment when Sophocles has Antigone upbraid the Theban king Creon, telling him that his rules were subordinate to the laws of God.

This is wishful thinking, as illusory and illogical as the belief in the inevitability of progress. Antigone is not talking about human rights, or notions which Western philosophy developed as natural law, but about divine law. Human rights themselves are entirely alien to the laws of every ancient civilisation. They are a notion developed in Western philosophy; a product of the Enlightenment, emerging against the background of the American Revolutionary War and the French Revolution.

And despite the genius of their expression in, for example, the French National Assembly's Declaration of the Rights of Man, or by Thomas Paine, they are fundamentally aspirational and utopian. They describe how the world ought to be in language which suggests that they are the ­natural order of the world.

But, as we can all see, no matter how much we might like the world to operate in accordance with, for example, the ECHR, in fact it does not, never has done and, unless and until the Kingdom of God is built on Earth, probably never will. That does ­nothing to diminish the desirability of the aims of rights, but it suggests they are badly expressed.

Rousseau's opening to The Social Contract - "Man is born free and everywhere he is in chains" - is a magnificent line, but as de Maistre pointed out, it makes about as much sense as saying "sheep are born carnivorous and everywhere eat grass". Thomas Jefferson's truths are similarly far from "self-evident", no matter how much we would like them to be, when measured against the laws of the universe, while his "unalienable" rights - life, liberty and the pursuit of happiness - are ­everywhere, and all too easily, alienated.

Yet the logic of rights is that, when they are infringed, our protests and calls for redress should be directed to the universe at large. The UN, the European Court of Human Rights and even Mr Starmer understandably struggle to fill those shoes. We might make a more successful attempt to make the admirable aims of rights a reality if we called them what they really are - duties. And duties which every one of us, individually, has an obligation not to infringe.