The former Cabinet minister's stint at the top of the BBC's governing body, which started in 2011 and was due to end next April, has seen him work with three different director-generals and weather scandals including excessive executive pay and the corporation's disastrous Diamond Jubilee coverage.
Lord Patten, who had heart surgery seven years ago, said he underwent further surgery last month after being admitted to hospital in London with "serious chest pains".
He paid tribute to the medical staff who treated him, saying: "On the advice of my doctors, however, and having consulted my family and friends, I have concluded that I cannot continue to work at the same full pace as I have done to date, and that I should reduce the range of roles I undertake.
"On this basis I have decided with great regret to step down from the most demanding of my roles - that of chairman of the BBC Trust.
His departure comes as the BBC faces further fallout from the Dame Janet Smith review into the corporation's "culture and practices" while disgraced DJ Jimmy Savile worked there. That scandal and the BBC's reaction led to the resignation of director-general George Entwistle - seen as a close ally of Lord Patten.
In a statement, Lord Patten said: "It has been a privilege to have served as chairman of the BBC Trust. Like the NHS, the BBC is a huge national asset which is part of the everyday fabric of our lives.
"It is not perfect - what institution is? It always needs to challenge itself to improve. But it is a precious and wonderful thing, a hugely positive influence which benefits greatly from the creativity and dedication of its staff."
One of Lord Patten's most fierce critics Philip Davies MP, said: "Always wanted to see Lord Patten leave the BBC Trust but not in these circumstances. I wish him well with his recovery... "