However, Justine Greening, the International Development Secretary, who has come in for criticism over the West Coast Mainline franchise fiasco because she was Transport Secretary at the time the deal with Aberdeen-based First Group was agreed, is expected to turn up, according to Conservative Central Office.
Mr Mitchell, who faced calls to resign when it emerged he had allegedly called police officers on duty in Downing Street "f****** plebs", has apologised to the officers concerned.
However, while he accepted he did not treat the officers with the respect they deserved, he still denies using the words attributed to him.
A friend of the Cabinet Minister said he had informed Prime Minister David Cameron he would not be attending the con-ference, which starts on Sunday, as he feared his presence would be a distraction.
"At the moment he is planning not to go to conference," the friend said.
"It was his own decision. He doesn't want to be a distraction."
Although Mr Mitchell had not been expected to speak at conference, it is highly unusual for the party Chief Whip to stay away altogether.
However, there is little doubt his presence at the annual get-together would have been a constant distraction as the media would have followed him everywhere.
The Chief Whip has agreed to meet Police Federation officials later this month.
Labour MP Tom Watson said of Andrew Mitchell's non-attendance at Tory Conference: "Andrew Mitchell is running scared.
"It's ridiculous the Chief Whip would rather dodge his own party conference, on the doorstep of his Sutton Coldfield constituency, than do what is right.
"This Government will blame everyone but itself for their failure. David Cameron should get a grip and make Andrew Mitchell come clean about what he actually said to police officers doing their job and apologise properly."