SINCE I mentioned the general unaffordability of the welfare state last week, there has been a fair bit of discussion on the topic.

Tabloid newspapers, the Prime Minister and the Chancellor of the Exchequer have all given us their thoughts on the topic which, as so often with opinions from these quarters, have served only to muddy the waters.

Don't get me wrong. I persist in my view that the level of welfare spending is unsupportable. Not that – despite the best efforts of hang-wringing Lefties and the BBC – it's a particularly contentious belief. As it turns out, surveys suggest that three-quarters of the population (including a majority of Labour voters) agree that we spend too much on welfare. All the same, I have a few qualms about this otherwise cheering consensus because I fear that many of these newly convinced critics of the welfare state are labouring under a misapprehension.

To judge by the general tone of the last week's commentary on welfare, you'd imagine that the main problem with welfare spending was families in which no-one has ever worked and which produced dozens of children purely as a means by which to milk the state. The hysterical reaction to the appalling case of Mick Philpott, who was jailed for life for the manslaughter of six of his children, appeared to suggest that it was also responsible for fostering criminality. That is nonsense; the case tells us nothing about the welfare system or, indeed, anything else other than the human capacity for wickedness.

All the same, there's no denying that the fact that the system can allow some people who could work to fall into dependency, at the expense of those who genuinely require support, is a significant objection to the welfare state, on both moral and practical grounds. But it is emphatically not the problem with welfare spending.

Even if one could, without errors, identify every single claimant who fell into the sponging, skiving or fraudulent categories and cut off their payments, it would still be an utterly piffling inroad into reducing the amount of money spent on the welfare state. In fact, if we were to abolish Jobseekers' Allowance itself entirely, it wouldn't make much difference in the grand scheme of things. It accounts for less than £5 billion of the total spent on benefits, which is currently around £220 billion.

The real problem with welfare spending can be summed up in one word: pensioners. The state pension alone accounts for about half of the total benefits bill – and that is without totting up the other entitlements of those over retirement age. For example, Attendance Allowance, which helps with care costs for the over-65s, costs the taxpayer more than all the benefit spending on the unemployed.

Entirely unsurprisingly, very few politicians draw much attention to this point, because people who are indignant about the benefits bill are happy to see it cut from the workshy, but less keen to see it taken away from their granny (or themselves). The retired middle-class reader of the mid-market tabloids that fulminate against "leeches" and "parasites" does not, on the whole, care to be reminded that most of the largesse being doled out by the state is being spent on, well, him or her.

The very word "pensioner" still, for most of us, conjures up a sense of the generation which struggled through two world wars to bring about the peace and, until recently, prosperity which those of us of working age now enjoy. But you could be in that group if you were born three years after the Second World War ended or even, if female, in the 1950s. A sizeable number of pensioners are younger than Mick Jagger and Keith Richards (both 69) – I mean, for crying out loud, Jet Black, the drummer with The Stranglers, has had a bus pass for almost a decade. These are the people who tuned in, turned on and dropped out. Hardly any of them fought in the Battle of Britain.

And of course, they are not The Few even without capital letters. Pensioners are an increasingly numerous class, as well as being, on average, much more prosperous than the young. What's more, a great many of them remain fit, lively and adventurous well into their 70s. Those of them who are middle-class homeowners are also those who have done best out of the welfare state – having received not only free university education, but grants to pursue it, benefited from the absurd rise in property prices and often in receipt of index-linked final salary occupational pensions. And in Scotland they can look forward to free care when the time comes, no matter how rich they are.

There are plenty of poor pensioners, too. But the resources of the benefits system should surely be directed specifically at those who are in real need, not matter what age they are, rather than handed out automatically just because you've passed some arbitrary birthday. It's easier politics, though, to direct indignation about welfare spending at spongers of working age, than to point out that someone who has paid off the mortgage on their big house in Pollokshields or Kelvinside and is in receipt of an occupational pension which might be twice the average wage doesn't really need the government to pay his bus fare. Nor, for that matter, the winter fuel allowance, if what you use it for is the air fare to your villa just outside Fuengirola, where you spend much of the year on the golf course.

The problem is that, for the Conservatives, it would be electoral suicide to target this group, while much of the Labour Party is (completely illogically, given its commitment to "progressive" taxation) ideologically opposed to means testing. There was the fleeting hope, more than a decade ago, that Tony Blair's Government might do something about the impending pensions black hole, but it was quickly dashed when he sacked Frank Field, the one politician who might have made a decent fist of sorting out these problems.

When Mr Cameron became Prime Minister, he gave Mr Field a job as the Government's "poverty czar". It would be nice if, in that capacity, he pointed out that we could do a bit more for the poor if we stopped hosing down people who are already very well off with public money, just on the basis that they can remember buying Sgt. Pepper's Lonely Hearts Club Band when it first came out, but I suppose it is too much to hope for.