Labour has renewed demands on David Cameron to axe him after a "clear the air" meeting with local police representatives ended with them saying his position was "untenable".
Mr Grayling described the incident as "very unfortunate", but he argued it was "a mistake for the Police Federation to be trying to use this in the way that they are".
Emerging from Friday's meeting in Mr Mitchell's Sutton Coldfield constituency, the chairman of the West Mercia federation Ken Mackaill accused the senior politician of questioning police integrity.
Although he admits swearing when he was refused access through the Downing Street gates on his bicycle, and has apologised to the officer involved, Mr Mitchell denies calling the officers who bore the brunt of his outburst "plebs".
That is at odds with the officer's written log of the outburst.
Mr Grayling said: "It's clearly very unfortunate.
"His conduct was un-acceptable, he's accepted that, he's apologised. I think really the matter should be left there.
"I think it's a mistake for the Police Federation to be trying to use this in the way that they are."
Defence Secretary Philip Hammond has also rallied to his Cabinet colleague's defence, accusing the Police Federation of trying to "hijack" the row to bring up other issues and grievances with the UK Government.