Once upon a time, you could hardly open a newspaper without someone on the letters page fulminating about the "misappropriation and perversion" of that good old English word "gay".

That these correspondents seldom felt the same way about good old English words like "queer" and "pansy" was, well, queer, but it was chiefly odd they were so interested in – indeed, in some cases, suspiciously obsessive about – the subject.

Like most people, I didn't have much interest in David Cameron's proposed law introducing gay marriage in England and Wales, which starts its progress through the Commons this week. It doesn't affect Scotland (though similar legislation is almost certainly on the way here, too), but it has become a hot topic because it has prompted Tory infighting on a scale not seen since - actually, scrap that. There is always bitter Tory infighting about some issue or another. This is merely this week's excuse.

One reason I wasn't very interested can be summed up by a poster pasted around New York at the moment which says: "If you don't like gay marriage, don't get gay married". Besides, in the words of Fred W Leigh, my wife won't let me.

The vast majority of the heterosexual population share this indifference. What puzzles me, as I say, is the small minority (and smaller than it used to be) of people who profess not to be tempted by homosexual activity, but who make it their life's mission to peer into other people's bedrooms. No doubt Freud had opinions on the tendency.

But now there's a new minority which puzzles me even more. It is a vanishingly small minority; almost a minority of one – to wit, the Prime Minister. This minority is staking his whole political position on a change which is, in secular terms, hardly a change at all, and for which practically no-one, gay or straight, was clamouring. Yet it is wildly unpopular with many of his supporters and stirs up all sorts of unintended complications.

The Labour Government in 2004 introduced legislation which extended the secular and legal provisions of marriage – everything, bar the name – to same-sex partnerships. Since consenting homosexual acts had been legal for almost 40 years in England and Wales (and almost 25 in Scotland), it was a long-overdue reform which allowed things such as the right to be listed as next of kin, or to inherit tenancies, pension benefits and the like, which were enjoyed by couples in civil marriages.

Since it equalised civil marriages and gay civil partnerships, it didn't have any impact upon religious groups, which were free to regard them as exactly as valid (or invalid) as registry office weddings. Liberal churches and synagogues – and I suppose in theory mosques, though I know of none quite that liberal – could bless such unions, and traditionalist ones refuse to bless them.

The only gay person I can remember being against the legislation was the heroic Peter Tatchell, whose obsessive campaigning has at least always been consistent, even when he seems to almost everyone else to be a slightly bonkers voice in the wilderness. He argued for gay marriage on the lines being proposed by Mr Cameron, but also for heterosexual couples to be allowed to sign up to civil partnerships, rather than getting married. The latter is not on offer in the legislation now being brought forward, so the notion that it introduces strict equality is inaccurate, at least for straight couples.

The proposed legislation is, as far as the state and the legal rights of individuals are concerned, about no more than a name. But that is enough to have created problems, because England has an established church. So the Tories have tweaked the legislation so it will be illegal for the Church of England to perform gay marriages – though they didn't bother to ask whether it wanted to or not (many C of E priests probably would like to). Far from protecting the churches from getting sued for discrimination, this makes it more or less inevitable at some point in the future.

No doubt a few people feel very strongly about this. But the force of the opposition to Mr Cameron, I suspect, stems more from the fact he has made the issue a priority, rather than because opposing it is a priority for all but those tiny few.

I have some sympathy for the Prime Minister's view that it's the right thing to do. But practically no one but him and Mr Tatchell had this high on their list of wrongs to be righted, or inequalities to be addressed. If he thought it would do something to show that Tories are not illiberal or opposed to minority groups from whom they have struggled to obtain support, it has spectacularly backfired.

Many aspects of the grassroots rebellion, and other leadership challengers, whether the improbable Adam Afriye or the altogether more threatening (from Mr Cameron's point of view) figure of Boris Johnson, have sprung up because rank-and-file Tories see the Prime Minister's position on the issue as indicative of skewed priorities. It is a metropolitan, liberal interest which doesn't matter to most people – gay or straight – nearly as much as more basic concerns.

Politically, they see it as every bit as daft as Nick Clegg selecting reform of the House of Lords or a poll on AV as his price for joining the Coalition, rather than choosing more popular measures such as fulfilling his pledge on tuition fees.

There are many other inequalities the Government might have chosen to address, and which Conservatives would have been delighted to challenge. Reform of the tax and benefits system, which has a marginal effect on the poorest people in society, say, or doing something about the extortionate price of fuel (which a report last week concluded was almost entirely due to taxes), with its catastrophic impact on rural communities, food prices and almost everything else.

But instead of tackling popular issues, Mr Cameron has chosen to concentrate on the dinner party topics of Notting Hill, where saving the planet, gay rights and liberal interventions in foreign countries are a big deal, but the subject of whether you can afford to fill the car, or buy the children new shoes, doesn't pop up that often. Even many of those who don't mind gay marriage find it baffling that a Tory leader should concentrate on that issue at a time like this. It's not so much the policy's unpopularity is damaging Mr Cameron, more his placing it above bread-and-butter issues is evidence that he's out of touch.