The London 2012 supremo, who is chairman of the British Olympic Association, was reportedly the Government's preferred candidate for the role but told the Daily Mail he did not have the "capacity" for the job.
The 1980 and 1984 1500-metre Olympic champion said: "I did allow my name to go forward to give myself time to properly analyse whether I had enough time to do the job to the best of my abilities.
"On reflection, I haven't the capacity and I now want to concentrate on my current commitments and the International Association of Athletics Federations (IAAF) election.
"As everyone knows, athletics is in my DNA."
In an interview with the BBC last month Lord Coe said he was "not sure" about the job.
Other names mentioned in relation to the role at the BBC's governing body include Dame Marjorie Scardino, the former chief executive of the company behind the Financial Times, and Channel 4 chairman Lord Burns.
Lord Patten stood down in May for health reasons after a turbulent three years in the job.
Former BBC executive Roger Mosey, who masterminded the corporation's Olympics coverage, welcomed the news when the Tory peer emerged as a possible replacement but said reform was needed.
Mr Mosey, whose career at the BBC included stints as editor of the Today programme, controller of Radio 5 live and head of television news, said: ''Seb Coe would be a great choice as BBC chairman. But the Trust role as now constituted is undoable: can't be both regulator and cheerleader.''
The BBC Trust, as governing body of the corporation, upholds standards and controls licence fee revenue.