“TELL us,” asked Ed Milliband, on his first outing on BBC's Question Time since he ceased being Labour leader, “which non-EU country would the UK be like post-Brexit?” The prominent pro-Leave Tory MP David Davis replied: “Great Britain.” The audience roared its approval. It was one of most telling exchanges in the EU referendum debate so far and must have sent a chill into David Cameron's soul.

After weeks of battering voters senseless with statistics claiming that Britons would be materially worse off it they vote to leave Europe, here finally was the essence of the referendum issue: it is about national autonomy, about self-government. Just as in Scotland, nationalism remains the most divisive issue in politics – it's just a different nation we're talking about. Europe for many English Conservatives is an existential question. And for them, country comes before party.

Already the sense of betrayal runs deep. Also on the Question Time panel was the former Cameron adviser, Steve Hilton, characteristically attired in a Silicon Valley track suit and trainers. Earlier he had outed David Cameron as a closet Brexiteer, saying that in private he'd always taken the Tory PM to be an “outer”. Cameron has denied that he is anti-Europe but I can well believe it. As with Scottish Nationalists attacking “Westmonster”, attacking Brussels has always been an easy way to appeal to the Tory base.

There are hardly any Europhiles left in the party. Most English Conservative constituency associations are either virulently anti-EU or just intensely Eurosceptic. All Tory leaders have to emulate Margaret Thatcher's visceral hostility to some extent. When Tory members heard Cameron promise, only a year ago, that he “could lead the Leave campaign” if his renegotiations failed radically to alter Britain's relationship with Europe, many took him at his word. He didn't sound like a pro-European then, talking of restoring “British sovereignty". But now they see him schmoozing Angela Merkel and the IMF's Christine Lagarde at the G7 in Japan, and regarding it as self-evident that leaving Europe would not just damage the British economy but could hold back world trade.

The Prime Minister says British families would be £4,300 worse off, house prices would fall 18 per cent, there would be a £20-£40bn black hole in the nation's finances, 820,000 jobs would be at risk, and pension pots could shrink by £1,900. These suspiciously precise figures come from the Treasury and nominally neutral think tanks such as the Institute for Fiscal Studies. But anti-European MPs like John Redwood and Jacob Rees-Mogg are furious at their leader’s doomy forecasts, which they believe are Brussels-inspired and unreliable.

It makes Cameron's earlier anti-European protestations sound hollow and insincere. If the PM really believes Brexit would bring economic apocalypse, would he seriously have led Britain out of Europe if he hadn't got his way on restricting benefits to EU migrants? Of course not. And of course, he didn’t even get his way on that. The whole renegotiation now looks to have been an exercise in window dressing, a faux crisis that had been orchestrated by the Brussels bureaucracy to make it look as if Cameron was striking a hard bargain and making them sweat.

Many Tories now suspect that Cameron was stringing them along and always intended to sell Britain out. “It's the biggest stitch-up since the Bayeux Tapestry,” says his arch rival, the former London Mayor, Boris Johnson. And the sense of betrayal is turbocharged by the latest immigration figures, showing that net 330,000 migrants, half of them from EU countries, entered Britain last year, the second highest on record. This is after Cameron promised repeatedly to limit migration to the “10s of thousands”. These figures indicate either gross incompetence or a lack of will to curb immigration – or both.

Europe and immigration are very closely linked in the Tory imagination. Both are regarded as existential threats to British culture and nationhood. It is a fear that England is becoming unrecognisable to its own inhabitants. We find this hard to understand in monocultural Scotland, but if you enter cities like Coventry or Bradford on any busy Saturday, white people seem to be in a minority. They aren't of course; it's just that most of the whites live out in the suburbs. But the visibility of Asian and Afro-Caribbean faces in English inner cities fuels the anxiety among many Tory, and many Labour voters, that national boundaries have somehow been breached and that there is a foreign invasion. That the British way of life is being radically changed without their agreement.

In this age of identity politics, and the protection of minority rights, white working-class Britons feel they are regarded as cultural junk, not worth protecting. EU migrants are of course also mostly white, but that doesn't alter the sense of grievance and alienation. There is a widespread suspicion among English voters that well-paid work is being stolen by migrants from countries like Romania, where the average wage is less than the UK minimum wage. Eurosceptic Tories such as David Davis now call the EU a “job transfer machine” designed to give British jobs to people from Eurozone countries which are no longer capable of generating them.

Immigration is likely to dominate the final stages of this referendum campaign, and it isn't going to be pretty. The sense of national decline and alienation that has fuelled the rise of Donald Trump in America, is abroad in England's green and pleasant land. Immigration is seen by many Tories as a national emergency and some believe Cameron is standing idly by while the country is lost. The Office for National Statistics has forecast that another 4.1 million migrants will arrive in England over the next decade – equivalent to four new Birminghams.

If Britain votes to remain (possibly on the strength of Scottish votes), there will be hell to pay in the Tory party. Already, the 70-odd Leave Tories are in a ferment and there is talk of a vote of no confidence in their leader. Boris Johnson has of course positioned himself for this eventuality by declaring himself for Brexit. He is now touring the country, battling with Remain protesters in gorilla suits, and generally stirring up the reverse of apathy. With his Brexit bus emblazoned with the bogus claim that membership of the EU costs Britain £350m a week, he could almost be on his own presidential campaign tour.

If Brexit fails, Boris hopes to benefit from the disappointment and resentment that has built up against Cameron. If Britain votes to leave he will be equally well placed. In the wake of a Brexit vote, huge efforts will no doubt be made by Brussels and David Cameron to keep Britain in some sort of continuing relationship to the EU. Britain is important to Europe and Brussels is unlikely to take no for an answer. As when Ireland voted against the Lisbon Treaty in 2008, concessions and opt-outs will be proposed with a view to another referendum somewhere down the line.

In this scenario, Cameron will appear almost as an agent of a foreign power to diehard Tory outers. The PM has already said that he will not stand again for leader of the party, so the campaign for the leadership of the Conservatives will effectively begin on June 24. The party is as divided as it has ever been – as divided as it was in the 19th century over the Corn Laws and free trade. And if it weren't for the fact that Labour is almost as divided as the Tories – though over different matters – then the 2020 general election would be there for the taking.