David Cameron is too much of a Tory toff to come to Scotland and debate for the Union.

He acknowledged as much in the House of Commons last month and, since then, he has allowed the notion to gather currency. The Prime Minister was amused when a Glasgow Labour MP suggested: "The last person Scots who support the No campaign want to have as their representative is a Tory toff from the Home Counties." Through laughter, Mr Cameron agreed that in Scotland "my appeal doesn't stretch to every single part".

I'm glad he found it amusing. I didn't. I can't see what his personal background has to do with the future of the country. Surely, in his role as Prime Minister, it is his duty to speak for everyone and to everyone.

Can you imagine a working-class Prime Minister agreeing that he shouldn't show his face in the Home Counties? Are we to accept there are no-go areas in the UK for politicians, depending on their socioeconomic origins? Is that how they think we judge them?

How very unmodern. Rightly or wrongly, the impression was confirmed when I saw Mr Cameron deliver a pro-Union speech from an empty, echoing Olympic stadium in East London. In the presence of a tiny clutch of people, he spoke to England Wales and Northern Ireland saying that the loss of Scotland would "rip the rug of our reputation from under our feet. The plain fact is we matter more in the world together. We would be deeply diminished without Scotland."

It was the first time I registered Mr Cameron expressing real regret, real loss at the prospect of Scotland separating. But I felt I was an eavesdropper. The speech came from the other end of the island and was addressed to everyone but us.

I can see that, coinciding with the start of the winter Olympics, the idea was to evoke togetherness with the golden memory of Chris Hoy and Andy Murray. In fact it offered the Nationalists another opportunity to say Mr Cameron was "feart" for refusing to debate with Alex Salmond. It gave substance to the notion that he felt on safer ground in the south.

It's an unfortunate impression that he has allowed to grow. People are repeating it. His Tory toff-dom is becoming established in people's minds as the reason he is the wrong person to lead the Union cause. And that is wrong, as wrong as it would be if he held back from arguing a case because he was black or gay or a woman. Instead of laughing about it and owning it, he should dismiss this toff stuff.

The Scots may not ever love him but, if he comes up to fight for the Union, if he chooses the right tone and makes the right argument in the right way, they will respect him. Mark Carney pulled it off. The Governor of the Bank of England might have anticipated a less than comfortable ride in Edinburgh. However, his speech hit the right pitch. He delivered his message with such elegance and diplomacy that he won his audience.

If the Prime Minister cannot do as much in every part of the UK he's not the man for the job, regardless of the bed he was born in. The stakes are too high for this class nonsense. We are living in a supposedly multicultural society that rewards merit and believes in equal opportunities for all. At least that is the stated aspiration. Yet this aspect of the independence debate would slot into the pages of The Beano circa 1960 with Lord Snooty versus the Bash Street Kids.

The argument goes that working class, west of Scotland men could hold the balance of power in the referendum. The Unionist fear is that they might vote for independence to punish the Westminster Government. I can see that they well might, unless Westminster offers a ray of hope. Like all voters, they will weigh which side offers most security and most opportunity.

Isn't the truth of the matter that Mr Cameron's problem is not his class, it's his character? It's not his paternity that will threaten the outcome, it's his policies. Scotland's concern is about his principles. What sort of a man is he at heart? Is he cutting welfare because economic conditions dictate that he must or would he want to slash it anyway?

Is he compassionate? Does he understand what a struggle people are having? Or is he a rattling can of theories who imagines the poor need less to survive on than people like him? What really makes him tick?

That's a hard thing to grasp long distance. The Cabinet is meeting in Scotland within the month. Let's hope he will take the opportunity to get better acquainted. And before he arrives, let's bury the myth that Scotland is hostile to toffs for being toffs. Does Tam Dalyell of the Binns attract hostility or warmth? On the other side of politics what about Lord James Douglas-Hamilton (now Lord Selkirk)? The man is universally popular. Throughout a successful career his background has been an irrelevance thanks to his unfailing modesty and courtesy. Tony Benn, who inherited and renounced the title Viscount Stansgate, can fill halls in Scotland. Toffs all three.

In the arts, Tilda Swinton is seen as a talent not a toff (though she was born one). Robbie Coltrane is rarely, if ever, associated with his Glenalmond schooldays. The writer and journalist Neal Ascherson is admired as left leaning with Nationalist sympathies. His Eton and Cambridge background is irrelevant. His intellect, rightly, is regarded as more important.

Even Mr Cameron would concede that he isn't as well connected as the Prince of Wales and the Princess Royal, both of whom are held with some affection in Scotland. But they earned it. They are regular visitors who give their time and influence to charitable causes. If the royals can do hearts and minds both sides of the Border, why can't the Prime Minister? At this crucial time in the history of the UK, why is he allowing his accident of birth to hinder his cause?

Isn't it the case that everyone has a disadvantage of some sort to overcome? Some people are born in poverty, some into a fractured or violent home. Mental or physical illness dogs others. Isn't the measure of the person their ability to transcend the difficulty, to put their disadvantage behind them whatever it might be? David Cameron's father overcame disability. He must have been so pleased to see his son born with only a surfeit of advantage to live down.

The question is: can Mr Cameron live it down for the Union, for Scotland? The issue surrounds a proposed debate between Alex Salmond and him. Some say that, by agreeing to the debate, the Prime Minister would elevate the First Minister to equal status in the eyes of viewers. I used to agree but I think the argument will look pretty threadbare on the morning after the referendum, if Scotland chooses independence.

I'd like to see Mr Cameron accept the challenge. He might lose the argument, though I doubt it. So long as he doesn't bray or hector, so long as he credits the electorate with the ability to see through his metaphorical top hat and tails, he might even win it. Either way, he will have earned our respect for taking the risk, toff or not.