At Labour’s Scottish conference in Dundee, a chastened Jeremy Corbyn told his battered troops what they didn’t want to hear: that the antisemitism row in the party could cost it the next election. It hangs over the party like a bad smell after another disastrous week of lurid publicity over Labour’s attitude to Jews.

The party is to be investigated by the Equalities and Human Rights Commission over allegations of “institutional racism”. For Labour to be accused by the very body set up by its own 2006 Equalities Act to tackle racism is an absolute disaster, even though the EHCR hasn't actually found Labour guilty.

Many activists, especially in Scotland, are understandably bewildered by this crisis, since they rightly insist that Labour is an avowedly anti-racist party, dedicated to opposing anti-semitism, and that they have never personally encountered any members who hate jews. Nor have I. The former Labour MP, Jim Sheridan, who is often cited as an antisemite because of a tweet he issued condemning Israel's activities in Gaza, was cleared by a party investigation. But his name still crops up on line.

Most of the cases that have been referred to Labour's disciplinary team are extreme anti-Zionism. There will be many more, since Labour has belatedly accepted the International Holocaust Remembrance Alliance's definition of anti-semitism as including any comparison of Israel to the Nazis, or accusations of genocide. Endorsing the IHRA is now beginning to look like an own goal. John McDonnell, the Shadow Chancellor, has himself accused Israel of “genocide”. Jeremy Corbyn has been allied with people who've accused Israel of behaving “like Nazis”.

Last week, Labour clearly hoped that that the spotlight would switch to the Tories after the suspension of 14 members over allegations of “Islamophobia”. One of them said “Islam is like alcoholism”. But whether or not these cases lead to expulsions - and not all will- Tory racism is never going to get the high media prominence that Labour antisemitism as received - and this is not just because of Tory bias in the press.

Of course, the Tories are riddled with both anti-semitism and islamophobia. It is in all the parties, to some degree. I remember when I was in the Westminster Lobby being told, by a senior Tory figure, that Malcolm Rifkind could never become Tory leader, “because he is a Jew”. I've heard dismissive remarks about Muslims. But there is a big difference between this casual racism and Labour's.

For a start, the Conservatives are a small party, with perhaps a fifth of the membership of Labour. Most Tories are old and they don't talk about their prejudices openly on social media, as Labour activists do, or in open party meetings. They confine it to pub chat in the Dog and Duck. Conservatives rarely appear on Twitter, which is a real problem for political balance.

Also, there is no internal dispute in the Tory party serving to amplify Islamophobia in the way the Israel Palestine dispute does in Labour. Most of the cases of jew-hating are down to Isreal-hating by Labour party members, who do not see condemning Zionism, ie Israeli nationalism, as anti-semitic. There is also evidence that a lot of phoney Labour supporters on Twitter are using the dispute to blacken the party's name.

Labour's antisemitism is not reducible to anti-Corbyn faction fighting either, but that is clearly a part of it. The “Tiggers" breakaway party, led by figures like the MP Luciana Berger, has used anti-semitism to condemn the Labour leadership. They are quite entitled to do this, of course, and Berger has been viciously attacked on line. But in the Tory Party no faction has an interest in digging out and promoting stories of antisemitism or islamophobia in the press. Baroness Warsi, the former Tory minister who accused the Conservatives of “institutional islamophobia”, is a relatively isolated figure.

Labour leapt on the Tory minister, Amber Rudd's, reference to Dianne Abbot as “coloured” last week, ironically amid remarks about abuse of women on Twitter. But this confusion of “coloured people”(bad) with “people of colour” (good) is an opaque racism to most voters, and Amber Rudd apologised for the confusion. The SNP deputy leader, John Swinney, fell into this linguistic trap himself some years ago – and frankly it is easily done, by people who are not racist. Rudd's utterance is not going to persuade many people that the Tory party is more racist than Labour.

It is not clear how Labour can extricate itself from the anti-semitism crisis – and it is now a crisis of the highest order. It has already damaged the party's image abroad and must be doing electoral damage at home. The row is creating widespread confusion and detracting from the party's broadly popular version of socialism. Labour's slippage further behind the Tories in the Scottish opinion polls last week, may be a reflection of this bad publicity.

Labour's Complaints Department even became a focus of antisemitism last week when it was announced that Laura Murray, an aide of Jeremy Corbyn, was going to oversee the investigations into anti-semitism. Labour later said this was not the case and the announcement had been made “in error”. But it does appear that Ms Murray has been seconded to the anti-semitism inquisition, much to the fury of anti-Corbyn MPs, and more seriously, Lord Falconer, who is himself going to lead an inquiry into Labour antisemitism. He said the appointment was “beyond mad”.

Then the Countdown presenter, Rachel Riley, who is jewish, announced that she intends to sue Ms Murray for accusing her of comparing Jeremy Corbyn to a Nazi. This was after last week's egging of the Labour leader outside a London Mosque. Riley had Tweeted “Good advice” over an old Tweet from left-wing commentator Owen Jones that said that if right-wingers don't want to be egged “don't be a Nazi”. Murray had replied that Riley's retweet suggested, wrongly, that Mr Corbyn was a Nazi too. I only refer to this as an example of how extreme and confusing the antisemitism row has become.

I have spent much of the week examining similarly confusing cases, often cited by Labour critics as clear evidence of anti-semitism in the party. Many are difficult to make sense of, and of course the vast majority are confidential so we'll never see them. Labour has investigated over 700 complaints in the last year of which the vast majority were found to be spurious or not from Labour members. So far, 96 Labour members have been suspended and 19 expelled. That is a very small number in a party of half a million.

The authority of the Labour Complaints process has been criticised by Labour's Deputy Leader, Tom Watson, as too slow and “not transparent”. The head of the Complaints Department, Jenny Formby, in turn condemned Watson for undermining her work. When party leaders and senior staffers are warring publicly like this, ordinary party members must be in despair. They should get their tin hats on, because it's only going to get worse.