The relationship began before Coulson became Brooks's deputy at the News of the World (NotW) in 2000 and continued for periods after he took over the helm and she moved to The Sun in 2003, the court heard.
In a letter Brooks wrote in February 2004 but ddi not send, she said she had been "waiting for six years" for him. But Coulson, a married father-of-three, said it was "not a wholly accurate representation" of events and that the affair ended around the time he left the Sunday tabloid in January 2007.
The pair are accused of being part of a conspiracy to hack phones at the now-defunct newspaper between October 2000 and August 2006. Under cross-examination, the former No 10 spin doctor told the court he had a different interpretation from his co-defendant.
He said the affair started in 1998 and stopped "soon after" but resumed around the end of 2003 or early 2004.
He said: "In between time, there was a very long period the relationship was what it should have been, which was friendship. From 2004 it was by no means continuous but the affair did continue until around the time I left the NotW."
Asked if Brooks's version was wrong, Coulson replied: "I'm not using the words completely false, it was a very complicated situation. People have different interpretations."
His wife Eloise was in court as he gave evidence for the seventh day at the Old Bailey. Earlier, the court heard that when he resigned over former royal editor Clive Goodman's phone hacking in 2007, he agreed a leaving package of £600,000.
The day his departure was announced, Goodman was jailed, along with private investigator Mulcaire, making him "effectively unemployable", said his lawyer David Spens, QC.
Coulson, 46, of Charing, Kent, and Brooks, 45, of Churchill, Oxfordshire, deny conspiring with others to hack phones. The also deny separate charges of conspiring to commit misconduct in public office. All seven defendants deny the charges against them.