'Ed Miliband thinks he's Benjamin Disraeli.

He talks of one nation. I've got news for Ed. You're no Benjamin Disraeli. Disraeli was a Conservative...'' Or so David Cameron can be expected to say in his leader's speech to the Conservative conference in Birmingham this week.

A Labour leader stealing Tory clothes is good copy for the Prime Minister. A sign, Cameron will say, that Labour has little to offer other than second-hand Conservative values with added public borrowing. Miliband being tough on immigration, welfare and public spending is also playing to Conservative themes, and as we know, on these issues the Tories always do it better. The Labour leader even paid homage to Cameron by emulating his without-notes conference address.

Strange, then, that there should be an unmistakable air of defeatism hanging over the Tory conference this week. "Cameron in Free Fall" declares the cover story on the conference issue of The Spectator magazine, home journal of the British Right. Inside, its editor, Fraser Nelson, does not mince his words. "By now", he writes, "it will be clear even to David Cameron that he is on course to lose the next general election". The economic recovery Nelson says is "evaporating" and Labour, having discovered that it has a leader is "cruising to power". Nelson is echoing the views of a sizeable chunk of the Tory party, including former heavyweights like the former defence secretary, Liam Fox, who will be putting the boot into Cameron this week.

An opinion poll in The Sun newspaper even suggested the PM reminds voters of a snake.

The Tories give the impression right now of being, if not the Nasty Party, then the Grumpy Party – fed up with immigration, multiculturalism, homosexuals, Europe, public spending, unions, Scotland – especially Scotland, following the Scottish Labour leader, Johann Lamont's, admission that it is the "something for nothing" society.

Above all, the they are grumpy about the Coalition with the Liberal Democrats, and loathe Nick Clegg and Vince Cable. Many Tories feel there has been too much pandering to LibDem obsessions like constitutional tinkering and gay marriage. They think that their Coalition partners are preventing the Tories from introducing deep cuts in public spending and deep cuts in personal taxation.

Now, I've been arguing for some time that the economy is going nowhere and the Coalition is doomed, but I didn't expect the Conservatives to agree so enthusiastically. I now find myself in the curious position of being somewhat less pessimistic than The Spectator about David Cameron's electoral prospects. I mean, are things really going that badly for him right now? Labour has a 10-point lead in the opinion polls – but that's nothing. It's quite normal for governments to have far worse ratings at this stage in the political cycle and still win come election time.

The Tories have just had a successful Olympics, as we will hear ad nauseam this week, even though it was really Labour who did the ground work when they were in office.

It's not as if Cameron is all that unpopular, either. His ratings are still better than Ed Miliband's on the economy and on who makes the most convincing prime minister – even after the Labour leader's polling bounce following his conference speech last week.

The omnishambles budget last spring certainly did damage to the Tory reputation for competence. And this week it became the omnishambles-on-wheels as the Government had to scrap the West Coast rail franchise. However, British voters are well used to shambolic behaviour from governments – we kind of expect it – and they don't appear to think that Ed Miliband would do very much better.

So, again: why do the Tories seem so down in the dumps? Well, I think it has something to do with the fact that the Conservative movement as a whole is feeling uncomfortable with itself right now. The Tory blogs are full of comments from longstanding party members who resent the "metrosexual" approach of the current leadership, and don't like David Cameron's promotion of "gay marriage".

Many Tories have felt locked out of their own party because of the leadership's determination to turn them into a touchy-feely, New Age Tory party that likes windmills, celebrates multiculturalism and hugs hoodies. Remember, the typical Conservative member is eligible for a bus pass.

Then there is Europe – the most divisive issue for the Tory Party since the Corn Laws. It is still causing trouble, even though the prospect of Britain adopting the euro is about as likely as Norman Tebbit entering a civil partnership with Abu Hamza. But the Tories simply cannot let the issue go.

No-one outside the United Kingdom Independence Party – which is now worrying Tory candidates in a number of English constituencies – seriously believes that Britain should withdraw from the EU, which remains the destination for half of Britain's exports.

This is the worst kind of issue to divide a party – a problem with no obvious resolution, an itch they cannot stop scratching even though they know it makes it worse. This narrow obsession with Europe only confirms the extent to which the Conservatives are out of touch with the real issues. It prevents them addressing the struggling voters – the "suspicious strivers" identified in a huge opinion survey unveiled by the Conservative chairman, Lord Ashcroft last week – who feel that this Government isn't on their side.

Cutting the top rate of tax from 50% to 45% made little economic difference, but has done serious damage to Cameron, because it confirms the suspicions of many hard-pressed families that he is only concerned with rewarding his rich friends.

So, the Tories are turning in on themselves, and losing touch with the concerns of ordinary voters. It is as if they are falling victim to the Labour disease of the 70s and 80s. This is good news certainly for Ed Miliband, who is leading a party that is more united and at ease with itself than at any time since its election victory in 1997. I don't really believe that he is "cruising to power" as The Spectator puts it, but he's certainly up and running. If the Tories continue to stumble, the Geek might actually make it to Number 10.