There is a smack of "no blacks or Irish" about David Cameron's immigration speech – an echo of the sign in a window that greeted me on my first summer in London.

"It doesn't mean your sort of Irish," the Earl's Court landlady explained when she found my sister and I retreating from her doorstep. And in truth I found the English to be the most welcoming of hosts and London to be a city of opportunity. There was no small country mind-set. No-one wanted to know which foot I kicked with or where I went to school. To secure an interview I didn't require introductions from a friend of a friend. It was a meritocracy. If I could do a job, it was mine.

For all that, the memory of that sign never left me. My wise old dad had forewarned me that while I was an individual in Ireland, outside it I would be seen as Irish. For all that I was born British, I was an incomer to London, so an immigrant of sorts.

I was lucky by comparison. I had my UK passport and an education. I was one of the skilled workforce for whom Mr Cameron is still rolling out the red carpet.

I pity those less fortunate in these times of austerity. The kindest thing I can think about the immigration initiatives Mr Cameron outlined in his Ipswich speech yesterday is that he was attempting to put a lid on a potential powder keg.

We have seen what happened in Greece. As the economy collapsed, the ugly face of fascism showed itself in the misnamed Golden Dawn party. Extreme-right activists are attacking dark-skinned foreigners and subjecting them to beatings with metal bars, bats and knives. They even have an emblem like a swastika.

I don't think it's too far-fetched to imagine something similar happening here if the economy sinks again. Was Mr Cameron attempting to signal his political control of immigration to stop an undercurrent festering into something more dangerous? The unkindest interpretation is that our Prime Minister is merely playing a dangerous game of party politics; trying to claw back right-wing Conservatives who are drifting to Ukip.

Certainly, his proposals have a hard edge to them. Some are couched in almost militaristic language. He talked of introducing private landlord checks on tenants and a crackdown on rogue businesses employing illegals on low wages saying: "We will undertake further targeted operations this summer, bringing together key enforcement bodies to form a series of local and national task forces."

Can you feel a net tightening? I can and it worries me. Immigrants are being sent a message that they will be charged for medical care, be low on the housing list and have no guarantee of unemployment benefit.

With net migration at 2.2 million between 1997 and 2009 and reaching a quarter of a million in 2010 alone there are parts of England that will breathe a sigh of relief that someone is getting a grip: showing leadership. For the rest (and that includes me) the speech is harder to take – especially when we read that foreigners occupy less than 10% of council housing. It is already the case under EU rules that those out of work lose unemployment benefit after six months. Also some estimate the cost to the NHS of treating people from the European Economic Area is marginal. When you consider that only two of the measures announced have start dates, it sounds like hot air. And that's unfortunate.

It's a shame Mr Cameron stooped to adopting a propagandist, prejudicial tone. I'd go further and say it was wrong of him. A Prime Minister should have better judgment and be more aware of the consequences of speaking carelessly on this most emotive of subjects. Everyone can see the need to control immigration – even Ed Miliband has conceded that Labour lost track of the inflow. It is also true that these hard times are going to get harder. People are complaining about cuts that have barely started. And hardship doesn't make people nicer.

It's in this climate that Mr Cameron is looking at immigration. Yes he acknowledged we need talent; we need the income from Chinese and Indian and other foreign students. Yes he previously acknowledged our need of skilled medics and competent tradesmen. The trouble is that immigrants are beating our lower skilled workers to available jobs. Boris Johnson pointed out on the Andrew Marr Show on Sunday that the problem is in our education system. Our under-educated and unskilled youth can't compete.

Mr Johnson said we let them down. Maybe we did and maybe they let themselves down too. Whatever the reason, some of them are bitter. Some are resentful. Too many, from a Tory perspective, are shifting to the welcoming embrace of Nigel Farage. Mr Cameron's politicking is unattractive, unnecessary and if it continues it will be borderline reckless. It risks branding all immigrants as "them" – a vulnerable category in any society. He needs to remain civilised in tone, be balanced and change legislation efficiently. Instead the Prime Minister has pumped up the issue to little real effect.

Desperate migrants will come anyway. The thoughtful, educated and talented may read his unwelcoming tone and take their money and their expertise to more congenial destinations.

And Scotland – which would welcome new blood and benefit from it – will be tarred by the Prime Minister's brush. Mr Cameron needs to see this is a real issue which needs a proper and sensitive debate. All he has done is underline the stereotype of the immigrant as a health tourist and a benefit scrounger. We have sizeable, long-standing ethnic minorities who are British. For their sakes too this issue needs to be handled with care. For their ongoing security Mr Cameron's job is to take the heat out of the immigration debate, not to stoke it.

Tub-thumping language reduces all to an unacceptable stereotype. It labels every new arrival and every long-standing, dark-skinned citizen with the reputation of the most badly behaved – when the truth is they contribute to and enrich this society. Most incomers just want the opportunity to build a brighter, better future – just like I did when I encountered that vile sign in London all those years ago.

If Mr Cameron doesn't know that he's in the wrong job.