In a live television discussion about the so-called "Plebgate" row, the Tory former Cabinet minister declared: "I have heard him use that word in private conversation - the pleb word, I mean."
That appeared to contradict Mr Mitchell's insistence at the height of the controversy over his row with Downing Street police officers that it was not the sort of word he would use.
Pressed on whether he had heard him use the word, Mr Portillo, who served with Mr Mitchell in John Major's Conservative government, told BBC1's This Week: "I think I did, but not in a bad context."
But the former politician - a regular pundit on the show - now says he has no recollection of hearing the word pass the lips of his former colleague.
"I seem to have misspoken," he said after his comments were reported. "I had no right to say that."
Mr Mitchell admits he launched into a foul-mouthed rant at police officers who said he could not ride his bicycle through the main Downing Street gate, but denies their claims that he called them "plebs".