The 74-year-old's decision brings an end a front bench career stretching back to 1972 as David Cameron embarked upon a reshuffle of his team ahead of next year's general election.
Clarke spent four years in Mr Cameron's Coalition cabinet after the 2010 election.
He said in a letter to the PM: "I understand that you are proposing to re-shuffle the Government this week and I am writing to let you know that I would wish to retire from Ministerial office in your Government.
"I have greatly enjoyed my four years in your Cabinet and my spell in the National Security Council. I have been heavily engaged in some parts of my portfolio such as the EU/US Trade Agreement, economic reform in Europe, support for small business, export finance, the fight against corruption, secure hearings in Courts and so on.
"However, I have just celebrated my 74th birthday and I have been doing red boxes at night for a high proportion of my adult life. There are plenty of other able people who could take on the work that I was doing in Government and I think the time has come to return to being a veteran back bencher.
He later added that he would be spending a lot of time watching cricket.
Mr Clarke added that he was "demob happy" for some time and had spent more of the past week at the England versus India cricket test match than in his office.
He told ITV News: "I shall carry on being MP for Rushcliffe. I'm afraid I'm a lifelong addict to politics."
Mr Clarke said he had enjoyed a "good long innings" in Government and would continue to be "largely supportive" of Mr Cameron from the back benches."
David Jones was sacked as Welsh secretary, universities minister David Willetts is understood to have stood down and declared his intention to quit as an MP next year while Nick Hurd said he was leaving his post as minister for civil society.
Conservative Big Beast Mr Clarke, 74, is stepping down from his role as minister without portfolio.
Former Welsh secretary Mr Jones said he had been asked to stand down by Mr Cameron, who is expected to take the opportunity to promote more women into key posts.
Mr Jones said: "The Prime Minister is carrying out a reshuffle. He is reshuffling the team and he has asked me to stand down. I totally understand his decision."
Mr Hurd announced his departure on Twitter, writing: "Am standing down having been given by DC the rare opportunity to do six years in a wonderful brief. Very proud of what we achieved.
"Thanks to so many friends and critics in our brilliant voluntary sector. You have often driven me nuts but my respect and love is undimmed."
A source close to Mr Willetts said he had "indicated a while back that he intended not to fight the next election".
Earlier Mr Clarke smiled and waved to photographers as he entered No 10 before news of his widely-expected exit from the Government was confirmed.
Former Tory leader Lord Howard of Lympne said: "Ken has made the most extraordinary contribution to our public life. In particular I think he was an outstanding chancellor of the exchequer."
He told BBC Radio 4's PM: "He always speaks his mind. We've been capable of maintaining a friendship over all these years despite the fact that we have quite often disagreed with each other on political issues.
"I think that says a great deal about the kind of person he is."
Alongside Mr Clarke, the Prime Minister is also expected to say goodbye to Cabinet veteran Sir George Young and promote a host of younger talent, including some of the stars of the 2010 intake.
Those tipped for advancement include employment minister Esther McVey - who was also seen entering No 10 - education minister Liz Truss and ministerial aide Penny Mordaunt.
Other MPs tipped for promotion include Margot James, Amber Rudd and Harriett Baldwin, as the Tory leader seeks to counter criticism that his government is still dominated by men.
Former defence secretary Liam Fox, a darling of the Tory right, could make a comeback to the political front line nearly three years after quitting in a row over his special adviser.