Coulson, 46, was given a job by David Cameron's party shortly after he resigned as editor of the Sunday newspaper over the hacking on his watch in January 2007, the Old Bailey heard.
But as allegations surrounding the tabloid resurfaced and media interest heightened years later, Coulson made a decision to leave his job at No 10 in January 2011.
In his evidence, Coulson, of Charing, Kent, told jurors he had been aware of one incident of hacking in 2004, in which chief reporter Neville Thurlbeck played a voicemail of home secretary David Blunkett declaring his love for Spectator publisher Kimberly Fortier.
Cross examining, prosecutor Andrew Edis QC said: "The position is if what you had done in relation to Mr Blunkett became public you would never have kept that job."
Coulson replied: "I cannot say for sure but it may well have meant I did not get the job.
"If I had explained what I explained to the jury now, that job would not have been offered to me."
His resignation as editor followed the conviction of royal editor Clive Goodman with private detective Glenn Mulcaire for hacking.
On the public announcement, Mr Edis asked Coulson: "You did not feel it was necessary to tell the truth to the public?
"You said you took responsibility for it because it happened on your watch which means 'I did not actually know about it but it happened on my watch, not my fault, but as a man of honour I will step down'."
Coulson replied: "I felt it was the right thing to do.
"I did not see myself as a man of honour."
He denied that he had "covered anything up".
Coulson denies conspiring to hack phones with Rebekah Brooks and others.
They both face separate charges of conspiring to commit misconduct in public office.
All seven defendants in the trial deny the charges against them.