Dozens of officers entered the politically sensitive area in Jerusalem's Old City to break up hundreds of protesters, Israeli police spokesman Micky Rosenfeld said.
He added that two policemen were slightly injured and 15 Palestinians arrested.
The mosque is part of a site revered by Jews as well as Muslims, and is a frequent source of friction.
It was not immediately clear what prompted this clash in particular. Israeli security forces had beefed up their presence in the area during the holiday of Rosh Hashanah, the Jewish new year, which began on Wednesday.
Israel captured East Jerusalem including the walled Old City in the 1967 Middle East war. It later annexed it as part of its capital in a move that is not recognised internationally. Palestinians want East Jerusalem as the capital of a state they seek in Gaza and the occupied West Bank.
Israel has said Jerusalem will remain its "indivisible and eternal" capital.
The two sides resumed US-brokered peace talks in July after a three-year stalemate, though neither has expressed much optimism for a major breakthrough. Earlier this week, a top aide to Palestinian President Mahmoud Abbas said the talks were proving pointless and would not bear fruit without much greater pressure from Washington.
Yasser Abed Rabbo said the negotiations, which started in late July after the three-year hiatus, had made no progress.