The Holocaust memorial in Berlin is immersive and disturbing.

Taking up an entire city square, it is composed simply of tall square pillars of smooth dark stone. Within, figures appear and disappear silently representing the millions who vanished under the Nazis.

By solemn coincidence, I was visiting the memorial on the very afternoon that Jewish hostages were being murdered by Islamic extremists in France. You wouldn't have known. There was no sense that home grown anti-Semitism was back in business in Europe. Even in the Jewish museum itself there were no obvious expressions of outrage or black arm bands.

The assumption is that, in Berlin at least, racialism is nowadays almost inconceivable. Berlin is founded on tolerance of everything - except intolerance. It's what everyone says here. In the rest of Germany, however, things aren't quite so harmonious. A new movement, Pegida, Patriotic Europeans against the Islamicisation of the West, has been holding mass demonstrations.

Last week in Dresden, up to 30,000 gathered to complain about immigration. The authorities in Berlin turned off the lights on the Brandenburg gate to express disapproval - disapproval of Pegida that is, not immigration.

Yet Pegida is less extreme than Britain's Ukip. It seeks only a tightening of the rules on the 200,000 asylum seekers who came here last year. In Britain, even 17,000 is seen as too much. These things are relative. David Cameron's call for an end to free movement in Europe is regarded as racist here.

However, there is a clash of intolerances which is causing some ideological dissonance among Berlin liberals. This week Pegida will be on the streets targeting, not immigration, but Islamic extremism, and wearing black arm bands in solidarity with millions in France who marched in defence of freedom of speech.

So keen are some liberals to distance themselves from Pegida that they are wondering if freedom of speech is always worth defending. You wouldn't tolerate anti-Semitic cartoons that offended Jewish people, except in a Jewish museum, so why should Charlie Hebdo be allowed unlimited freedom to offend Muslims?

This echoes those remarks in the leaked emails from Al Jazeera journalists who warned that, if you offend 1.5 billion Muslims, then some of them might want to kill you. "What Charlie Hebdo did was not free speech" wrote the Doha-based journalist Mohamed Vall Salem, "it was an abuse of free speech". This attitude has been widely aired on twitter in the UK often by people on the liberal left.

It is quite wrong-headed and illiberal, however. The Hebdo cartoons were not hate speak. They were tested in court in 2006 and judged to be inoffensive satire. Moreover, Hebdo's cartoons poke fun at all religions equally: Christian, Jewish, Hindu. The trouble is, almost any representation of the prophet is seen as offensive by Islamic extremists.

We cannot allow religious obscurantists to abolish freedom of speech. I find it sad to hear Western liberals arguing that we have no right to represent faith in an unflattering light. This is creating, effectively, a law of blasphemy. For, if we cannot lampoon Islam, then how can cartoonists continue to represent Roman Catholics in this way?

Then, what would prevent ethnic minorities from insisting that they should not be represented in a way they find offensive? I don't particularly like jokes about Scots but I would defend the right of comedians to make them. Are we going to have the Guardian's Steve Bell arraigned in the court of public offence for that cartoon telling Scotland to go f*** itself, those grotesque portrayals of Alex Salmond in a kilt?

The Guardian implied that it had not reproduced the Hebdo cartoons because it considered them to be infantile. But their cartoonist Martin Rowson presents all manner of people in public life as sewage rats, including members of the Royal Family. Should the people who find that offensive have the right curb the cartoons?

We really cannot go down this road. Hate speak and racialism are already illegal; that is a diversion. If we proscribe images of religious icons because certain believers are offended then our concept of liberty is dead and buried. And it is the millions who died fighting fascism who are mocked.