Brussels was an atrocity waiting to happen. All the indicators pointed to an assault by jihadi terrorists who had pinpointed the city as a soft target: it was not a case of if it would happen but when it would happen. In the aftermath of last week’s atrocities in the Belgian capital it became clear that the Islamic State (IS) had adopted “more aggressive” tactics in its campaign against the west and that up to 400 jihadi fighters had been specially trained to carry out attacks against targets in western Europe. Special camps have been established in Syria and northern Iraq where male and female fighters are schooled in urban terrorism, using tactics designed not only to kill civilians but also to create the maximum chaos. Last week’s double-pronged attack in Brussels did just that, leaving 31 people dead and an estimated 270 wounded. Three suicide bombers also died while carrying out their assault.

The head of the EU police agency Europol Rob Wainwright warned starkly that in the months ahead the number of assaults could grow and that many of those involved could come from the 5000 Europe-based fighters who have travelled to join the fighting in Syria and are now gradually returning home. “We are faced by a more dangerous, a more urgent security threat from so-called Islamic State,” Wainwright said. “It threatens not just France and Belgium but a number of European countries at the same time...It is certainly the most serious threat we have faced in at least a decade.”

It is not difficult to see why European cities have become such soft targets. Most are large and open conurbations with good transport infrastructures and diverse populations, many of whom come from Islamic backgrounds. However, the very attributes which make them popular tourist destinations also make them vulnerable. With the tourism season getting into full swing large crowds in public places are commonplace at this time of year and are almost impossible to defend against the kind of bombing attacks which were launched across Brussels last week. There is a desperate synergy in the fact that the very aspects which make these cities attractive provide the terrorists with easy pickings.

Other factors are the relatively open borders of the Schengen visa-free legislation which allow people with EU passports to travel across the continent virtually unimpeded. Add on Europe’s mainly liberal approach to governance and its well-advertised dislike of encroaching on civil liberties and there is much for IS to exploit when deciding on western Europe as a suitable target. Belgium offered other advantages. It is very much a divided society in which Flemings and Walloons lead almost compartmentalised lives with separate languages and means of communication. Almost every aspect of Belgian life reflects this division which is not only on linguistic lines but runs through all levels of society. By no means is this an artificial division: it influences education, the media and politics and its one of the reasons why there has been a chronic weakening of national identity and a lack of certainty about the whole sense of being Belgian and what that entails in terms of belonging.

The country also contains recognised enclaves with large numbers of disaffected and alienated young Muslims who suffer from poor social and economic conditions and easily fall prey to criminal influences. Molenbeek, home to the Belgian terrorist cell was just such a place - with the added bonus that it is surrounded by more affluent areas which house the main institutions of the EU. Not only does Molenbeek area suffer from 40% youth unemployment but many of those youngsters are addicted to drugs and alcohol and perceive themselves as being an under-class which faces discrimination from white Belgian society. As a result crime is rife in Molenbeek and it comes as little surprise that the members of the cell had criminal records or had been in trouble with the police. For example one of the suspects Salah Abdeslam, arrest for his part in the November Paris massacres, and his brother Brahim - who was killed in the Paris attacks - used to run a cafe in Molenbeek that sold alcohol and was closed down for drug dealing offences. Although the brothers and their associates had been radicalised and supported the concept of holy war against the west none of them were interested in following the strict tenets of Islam.

Even so, with their tight network of friends and family and their shared grievances against the Belgian state they made ideal resources and were quickly snapped up by IS recruiters who played on their sense of alienation from mainstream Belgian society. According to Belgian senator Alain Destexhe this mixture of criminality and fundamentalism was a heady brew which was allowed to co-exist by a state which “tolerated them as long as the threshold of violence was not crossed”. It also has to be said that because Molenbeek became a no-go area for the police and security agencies it was often left to its own devices – one reason why Abdeslam was allowed to continue operating unhindered despite his alleged involvement in the Paris attacks last year.

The problem was intensified by the way in which the Belgian police and security services operate. In Brussels there are six local police departments to cover 19 municipalities containing 1.2 million people. If suspects feel that the police are getting too close or feel under pressure as one of the suicide bombers Ibrahim el-Bakraoui clearly did (according to messages to his mother found on his laptop) it was a simple matter to move from one police district to another to avoid capture. The divide in Belgian civil society also helped the terrorists as it has become clear that the security services were often loath to share information and intelligence due to the rivalries between various institutions. All this was common knowledge and would have been known to IS strategic planners before they decided to make Brussels a prime target.

Combatting the IS terrorist threat will not be an easy matter and it comes as little surprise that western security commanders are already discussing the problem in terms of the emergence of a new international conflict. In those terms this will not be a brief struggle but one that could last months, even years as the west attempts to find a response to what is becoming an intractable problem. That much became clear even before the first bomb exploded in Brussels airport when Belgian prime minister Charles Michel warned that an attack was “serious and imminent”. There are also concerns that the twin-pronged assault by the IS-sponsored bombers was neither spontaneous nor even well planned but was prompted by the arrest of Saleh Abdeslam, Belgian born but of Moroccan descent under arrest for the Paris atrocities last November.

In other words, ever since last year’s attacks on the French capital and the steady emergence of IS as an entity which has serious ambitions to extend its caliphate beyond its presence existence in northern Iraq and eastern Syria there are grounds for believing that the threat from extremist Islam is a clear and present danger to western democracies. How it can be countered is another matter. As became clear in Paris last year and in Brussels last week (not to mention Madrid in 2004 and London in 2007) it is not easy to stop suicide bombers intent on exploding their devices in crowded and unprotected areas with open access. As Brussels shows even airports with their plethora of security measures are not exempt from attacks of this kind carried out with very basic weapons – assault rifles, grenades and explosive belts or nail bombs packed into luggage.

Against that kind of armoury carried by a handful of determined terrorists who will stop at nothing to achieve their aims security forces are always going to be hobbled but there is a means of fighting back. As has been shown in the air campaign against IS in Syria and Iraq where drone operations have rapidly become the norm leading to the summary execution by missiles of top commanders the battle is not a one-way street. In those instances the west not only has the technology to carry out successful missions but it also has the necessary intelligence to plan them. This refers not just to communications interceptions though mobile telephones and email but also to human intelligence and it is already clear that western agencies have managed to place agents within IS structures – no easy matter considering the cultural and religious background and the fierce tribal loyalties of those involved.

In the fight against terrorism, intelligence gathering remains paramount – the ability to watch the opposition 24/7, to give them no hiding place and to ensure that their every movement is monitored and that nothing is missed. The public might complain about omni-present surveillance cameras in city streets but they play a vital role in the struggle. Getting the balance right is central to any counter-terrorism policy as it all too easy to use the notion of a national emergency to jeopardise individual freedoms and to endanger civil liberties on the pretext of protecting the state.

The restrictive Patriot Act in the US is a case in point – passed in the wake of the September 2001 attacks in New York and Washington this gave the administration extensive powers to act against further terrorist threats and included indefinite detention and the expanded use of unauthorised intelligence gathering. According to the American Civil Liberties Union there have been far too many instances when this law has been abused with the result that “while most Americans think it was created to catch terrorists, the Patriot Act actually turns regular citizens into suspects.” Amongst the controversial measures is the provision for issuing National Security Letters which permit FBI agents to monitor phone and email communications, collect bank and credit reporting records and track internet activity. Ten years after the passing of the act Senator Ron Wyden (Democrat, Oregon) warned that the dangers inherent in the legislation remained and that he would not be surprised if Americans would be “stunned and angry” when they discovered the extent of the infringements.

That much was recognised at a recent meeting of the Council of Europe when the Parliamentary Assembly’s rapporteur Pieter Omtzigt of the European People’s Party reported that a “surveillance-industrial complex” had begun to emerge in Europe. His conclusion was that the result has been the fostering of a culture of secrecy across the continent and that this was not necessarily a good thing: “These powerful structures risk escaping democratic control and accountability and threaten the free and open character of our societies.”

As Europe considers the best way of countering this latest threat many of the lessons from the war against terrorism are still obvious. The attacks in Paris and Brussels were not unexpected in that the threat level was high and western security agencies had sufficient intelligence to make them fear that an attack was in the offing even though the finer details were not uncovered. However, in both cases mistakes were made. In the aftermath of the bombing in Brussels Turkish President Recep Tayyip Erdogan revealed that last July his country’s security service had deported one of the attackers Ibrahim El Bakraoui to Europe. At the same time they warned their European counterparts that they believed that he was a militant but Ibrahim was later released by the Dutch police as there was no sign that he was involved in any kind of terrorist activity

This suggests a serious lapse by the European authorities but without hard evidence it would have been difficult for the Dutch to act – Ibrahim and his brother Khalid had no known extremist links until an apartment one of them rented was traced to Abdeslam last week. French Senator Nathalie Goulet, an expert in counter-jihadist operations, put the question into perspective when she said: “The reality is that if we knew exactly how many there were, it wouldn’t be happening.”