The cabinet minister said Clarkson's apologetic explanation that he had tried not to utter the n-word should be enough to draw a line under the controversy.
But David Cameron's official spokesman said the Prime Minister believed that his friend's fate was a matter for the broadcaster to decide.
Clarkson became embroiled in a racism row following claims that he used the word while reciting the nursery rhyme Eeny, Meeny, Miny Moe during shooting an episode of the BBC2 motoring programme.
In the footage Clarkson is using the nursery rhyme to compare two sports cars.
He said he "mumbled where the offensive word would normally occur" in two takes, and then used the word "teacher" in its place in a third instance.
Last night he posted a video in which he said he "did everything in my power to not use that word" and was now "begging your forgiveness for the fact that obviously my efforts weren't quite good enough".
Asked if he backed calls for Mr Clarkson, well known for courting controversy, to be axed over the latest episode, Mr Gove told ITV's Good Morning Britain: "No, I don't." He added: "He has been clear in his apology and I think we should leave matters there."
Last night deputy Labour leader Harriet Harman called for the BBC to sack the presenter.