He could have spent the next three years fending off headlines about sibling rivalry, or he could take his "dream job" in America where only a handful of anoraks even know who he is.

It's not difficult to see why David Miliband has chosen to join a humanitarian charity in New York. Not since Frasier and Niles Crane have two bookish siblings in the same profession been watched so avidly in the hope they will do each other down. From where the elder Miliband is standing, it just isn't funny any more.

David's departure doesn't mean, of course, that he will never be back. He may feel it wise graciously to leave the orbit of British politics until such time as Ed's star has burned itself out, and then make a comeback. If so, he will have done himself no harm to have worked in a meaningful role delivering vitally important services to people in need. It's just a pity so few of his former Westminster cronies have done the same.

Neither the Prime Minister nor David's brother Ed have had much of a career outside politics. Both did politics, philosophy and economics (PPE) at Oxford, the closest there is to a foundation course for politicians-in-waiting. After a brief spell as a TV journalist, working – you guessed it – on a politics programme, Ed started working for the Labour Party and became an adviser to Gordon Brown, being elected an MP in 2005. David Cameron, meanwhile, joined the Conservative Research Department straight from university before becoming an adviser to Norman Lamont and Michael Howard. His seven years as a PR man – working for media company Carlton Communications – at least gave him up-front experience of business, but wasn't exactly at the sharp end of service delivery.

The ideal for any party should be to have elected representatives from all walks of life. Why? Because politics is not the real world as most people experience it – you only have to listen to the frat-house din of Prime Minister's or First Minister's questions to understand that – and it is already too easy, particularly at Westminster, for politicians in that boarding school atmosphere to lose a sense of reality, becoming more like each other than they are like the people they serve.

Those whose professional experience and sense of identity is rooted in a sphere outside politics, be it teaching, manufacturing, nursing, running a small business, doing a trade, or working in the creative industries, are more likely than a party apparatchik to understand the challenges facing others working in that sphere; that just stands to reason. If nothing else, these political outsiders tend to have a warmer reception from the public than career politicians because people see them as one of their own. Think of former Labour Health Minister Alan Johnson (a former postman), former Scottish Justice Minister Cathy Jamieson (a social worker), and former ship steward John Prescott. Working class Prescott held on as Deputy Prime Minister for 10 years at least in part because of his popularity with the Labour grassroots, who were put off by former lawyers and media men like Tony Blair and Peter Mandelson.

There are many former lawyers at Holyrood and Westminster, Nicola Sturgeon and Kenny MacAskill among them. A slew of politicians have also dabbled in journalism, such as Ed Balls, but these professions have always been on a two-way street with politics. Truly representative parliamentary politics requires a broader cross-section and on that score, Holyrood perhaps does better than Westminster. The two most recent first ministers of Scotland have been a former maths teacher (Jack McConnell) and an economist (Alex Salmond) and on the backbenches, there is a wealth of professional experience, from social workers such as the SNP's Dennis Robertson to doctors like Labour's Richard Simpson, a GP and psychiatrist.

So, for David Miliband, a spell doing humanitarian work would not only be worthwhile and fulfilling but a CV enhancement should he ever wish to apply for the job of Prime Minister. The problem is, having left politics, few former politicians go back. Pity – it would do politics good if they did.