IT'S a measure of how attitudes to Europe have changed in Britain over the last 20 years that anti-Europeans like Nigel Farage of UKIP, who was barracked in an Edinburgh pub last week, are now regarded almost as members of the political mainstream – in England, at least.

Yet, for people of my generation, it is hard to regard the European Union as anything other than a huge advance in European civilisation. I can remember when it was impossible to travel to Eastern European countries like Poland, Czechoslovakia and Latvia. They were communist dictatorships, closed societies, where often impoverished populations lived in fear of the state and had no human rights. They were part of a military alliance that threatened the very security of the West. Now these countries are vibrant European democracies and pose no threat to anyone. This has happened in only 25 years – the blink of an eye in historical terms.

Of course, Eurosceptics say this has more to do with Ryanair than the European Union; that these countries have become part of the European family simply because of the collapse of the Soviet Union and the expansion of capitalism. Conservatives like the MEP Daniel Hannan say that Europe itself today poses a threat to democracy because of its bureaucratic institutions and its lack of respect for diversity among the 27 member states. But no-one who recalls the fall of the Berlin Wall in 1990 could deny that Europe played a part in the democratisation of Europe. It is not just a willingness to host stag and hen parties that gets you into the European Union.

Countries like Bulgaria and Romania had to liberalise their societies in order to apply for membership under the Copenhagen Criteria established by the Maastricht Treaty in 1993. To be eligible, countries must not only have market economies, they must also have democratic institutions that observe human rights and respect the rule of law. These aren't empty phrases. Applicants must demonstrate that they have a free press, trades unions, independent judges, secret ballots and local democracy.

Some countries had to alter their constitutions. Romania, a state that had a poor record on human rights, had to recognise no fewer than 19 minorities and guarantee their political representation before it was allowed to join the EU.

When I visited Latvia during the 2008 financial crash, the minority Russian population in Riga was living in fear of victimisation. They had been the scapegoats in the past when the economy got into difficulties. But not this time – because Latvia was trying to qualify for membership of the eurozone, and the political leaders on both sides of the ethnic divide refrained from blaming each other for rising unemployment and falling wages. Amazingly, Latvia got through its difficulties, has the fastest growth rate in Europe and is now the poster child of the International Monetary Fund. But it was being part of Europe that made the difference.

And the civilising influence of Europe didn't just start in the 1990s with Maastricht. People in Britain are mystified at the determination of countries like Spain and Greece to remain in the EU despite having suffered internal devaluation, mass unemployment, a property crash and shrinking exports. Why didn't they just leave the EU, ditch the euro and devalue their currencies? Why remain under the heel of the European Central Bank (ECB)?

But you have to remember that as recently as the 1970s, both Greece and Spain were fascist dictatorships. Spain was under the rule of Franco until 1975, and there was an attempted fascist coup as recently as 1981. These countries feel that being part of Europe has brought them into the modern age, and provided security against any reversion to totalitarianism. But not only that. They have been transformed by European trade and see no viable economic alternative to being part of the world's biggest trading bloc. The EU has a single market of 500 million people, a GDP worth $16 trillion (£10.5tr) and the highest standard of living in the world.

Ireland suffered one of the worst sovereign debt crises of any eurozone country. When its banks crashed in 2008, this small nation bravely took full responsibility for the huge debts. It led to inflation, unemployment, and even the return of mass Irish emigration for the first time in a quarter of a century. Why did they stick with the euro and submit to austerity?

I was in Dublin on the weekend the euro was introduced. I expected to find deep reservations among citizens of this proud and fiercely independent country about relinquishing economic sovereignty. After all, they'd fought a civil war to leave the UK – why would they voluntarily hand their national destiny to Brussels? But I found there was genuine pride at the coming of the single currency. In pubs and taxis, people were holding up their euro notes and coins as if they'd won the lottery. Dubliners felt that finally they were the equals of Britain, of anyone in Europe. They were part of a great European movement and no longer backward, ex-colonial appendages to the UK.

Ireland benefited economically from Europe – initially from billions in aid under the structural fund, and then latterly from inward investment. American companies like Amazon, Apple, Cisco, Dell, eBay, Google and IBM have located in Ireland to be part of Europe and gain access to this huge free market. And they are still there, despite the banking crisis. When I returned to Dublin in 2008 at the height of the banking crash, I found civil servants accepting 30% reductions in pay with scarcely a murmur. There was no clamour to leave the euro, no resentment at the ECB and Brussels, still less the Germans.

It is proof of the tenacity with which countries cling to the ideal of Europe that they will go through severe hardship in order to remain part of the eurozone club. Europe is not just a matter of national advantage; it is a political and moral project that is greater than the sum of its debts. I suspect this is why Britain will never be a full member of the European Union.

When the European Economic Community (EEC) was first formed in 1957, it was explicitly designed to prevent war in Europe by harnessing France and Germany together in an economic block so tight that they would no longer be able to conceive of conflict. It worked. But Britain remained aloof, believing that, since it had won the war, it didn't need to be part of this forced economic integration. Moreover, Britain had the Empire and then the Commonwealth – a global economic network of mainly English-speaking countries with which to trade. Britain didn't think of itself as just another European country.

This delusion lasted until the 1960s, when Britain realised it was falling behind the country, Germany, that it was supposed to have beaten in two world wars. The European Community (EC), as it was known after 1963, was becoming a continental powerhouse, and industrialists and bankers in Britain started agitating for membership, arguing that splendid isolation was no longer viable. The Conservative prime minister, Edward Heath, finally took Britain into Europe in 1973.

Ironically, most of the running on Europe has been made by Conservatives. It was Margaret Thatcher who signed the 1987 Single European Act that created the single market, and she took Britain into the Exchange Rate Mechanism, despite her hostility to the social protections envisaged by Jacques Delors, the interventionist EU President.

The British left used to be opposed to the EEC, which it called "the bankers' Europe". In 1975, when Britain last voted for Europe in a referendum, Labour left-wingers like Tony Benn said it was a conspiracy against working people. The prospect of free movement of capital and labour across this huge economic space was a threatening one for trades unionists. There was a risk of "social dumping" – a race to the bottom as member countries abandoned worker rights in order to compete with low-wage countries where they didn't apply.

The Social Chapter of the Maastricht Treaty in 1993 was supposed to address this by giving all EU workers basic rights to things like maternity pay, a 48-hour working week, gender equality, holidays and so on. It is a pretty feeble document, which doesn't even formally include the right to strike. Nevertheless, it was considered so left-wing that neither John Major nor Tony Blair was prepared to sign up to it, and Britain still has an opt-out to the 48-hour week. These very basic social protections are now condemned by Conservatives as an alien imposition on British freedom. There are problems with bureaucracy in the EU, but these can only be addressed by giving the European Parliament more power, and Britain is bitterly opposed to that.

British Conservatives want to have access to the single market without having to observe the conditions of European Union membership – one of which is membership of the single currency. The sovereign debt crisis that broke in 2010 traumatised Europe, but the euro hasn't fallen apart as most commentators in Britain assumed it would. Instead, Europe is pressing ahead with deeper economic integration, and next year will introduce a wide range of banking reforms including a financial transactions tax (FTT), or "Tobin Tax" on transactions between banks. The idea is to get the banks to pay for any future bail-outs – which cost European taxpayers more than €4.6 trillion during the financial crisis. The European Parliament has also voted to cap banker bonuses to 100% of earnings.

Britain doesn't have to go along with this – only the eurozone countries will be subject to the FTT. But fearful of the impact on the City of London, the UK Government has been threatening legal action to prevent it happening anywhere. David Cameron and Boris Johnson have campaigned furiously against any cap on bankers' pay, even though it is supported by the vast majority of British voters. In 2011, David Cameron tried to veto new rules on financial regulation in Europe to put limits on future deficits. The eurozone countries ignored him, and went ahead.

Britain is already semi-detached from the euro and the Social Chapter and has refused to implement the Shengen scheme for ending border checks between EU states. It seems inconceivable that Britain will sign up to the moves towards further bank regulation and the creation of what is called a "fiscal Europe" in which member states will have to contribute to an EU treasury with the ability to issue bonds. This will be the parting of the ways. David Cameron says that if he cannot secure the repatriation of powers and fails to block banking reform, the Conservatives will hold a referendum advising withdrawal.

Tories say Britain will do better on its own, and be able to trade with the rest of the world instead of relying on Europe, despite the fact that it is the destination for more than half of British exports. But the rest of the world isn't so sure. Last week, Barack Obama urged David Cameron to stick with Europe, because America and the EU are about to sign a mutual trade pact that could be worth hundreds of billions in trade over the next decade. Britain still benefits hugely from Europe – one-quarter of all the inward investment that comes to Europe comes to Britain.

But it has never been a purely economic question. Europe remains the great ideal: it has created a new international political entity which is based not on borders and frontiers, but on freedom and diversity. In Britain, we are struggling to retain a sense of national exclusivity. We are supposed to condemn these nasty Europeans who want to steal our bankers' bonuses and flood Britain with immigrants. Well, when it comes to the crunch, I know which side I'll be on. And it won't be that of Mr Farage.