The fans are known to sing 'yid army' in reference to their links to the Jewish community in North London, but the term is deemed offensive and the English FA has asked them to stop or possibly face criminal charges.
However, Mr Cameron said that no hate crime was being committed as the term was not being used by the fans as an insult.
"You have to think of the mens rea. There's a difference between Spurs fans self-describing themselves as Yids and someone calling someone a Yid as an insult," Mr Cameron told the Jewish Chronicle.
David Baddiel, who is Jewish, has called on the Prime Minister to change his views, saying: "Yid is a race-hate word."
He said: "It was daubed across the East End by Oswald Moseley's Blackshirts, along with the word Out.
"The only possible reason why a culture that has tried to dismiss other race-hate words to the margins of language would consider it acceptable is if the racism of which it is a part is somehow less offensive, somehow less significant, than other racisms, which must be, I guess, what a lot of people consciously or unconsciously think.
"If it had been the N-word or the P-word, it wouldn't have got past David Cameron's lips."