Mr Cameron said the two leaders had agreed "an unprecedented level of co-operation" and would sign up to a strategic partnership between their two countries in the autumn.
At the same time, they also agreed to the opening of an office in the Qatari capital Doha for negotiations between the Taliban and the Afghan High Peace Council.
Mr Cameron said the agreement should send a clear message to the Taliban. "Now is the time for everyone to participate in a peaceful, political process in Afghanistan. This should lead to a future where all Afghans can participate peacefully in that country's political process."
Mr Karzai said he and his Pakistani counterpart had had a "very frank and open discussion" and echoed Mr Cameron's appeal to the Taliban to join peace talks.
The Afghan leader said he hoped relations with Pakistan, which have been strained in the past, could in future be "very close, brotherly and good neighbourly".
Mr Zardari said Pakistan would fully support any dialogue with the Taliban, whose fighters often operate from remote areas of his country.
He said: "Peace in Afghanistan is peace in Pakistan. We feel we can only survive together. We cannot change our neighbourhood or our neighbours."
A joint statement issued after the talks said Mr Karzai and Mr Zardari had agreed arrangements to strengthen co-ordination of Taliban detainee releases from Pakistani custody in support of the peace process.
The talks, in the secluded surrounds of the Prime Minister's official country retreat, were attended by senior military and intelligence figures including the UK Chief of the Defence Staff, General Sir David Richards, and MI6 chief Sir John Sawers.
With British and other foreign forces preparing to withdraw from Afghanistan next year, Mr Cameron said it was in all three countries' interest to achieve stability.
The Prime Minister said: "We share the same vision for Afghanistan – a secure, stable, democratic country that never again becomes a haven for terrorism.
"Clearly, there is much more work to be done in the months ahead and the United Kingdom will continue to stand firmly behind both countries as they work together to bring stability to the region."
Before the meeting, Mr Karzai said peace required the involvement in talks of "external elements involved in creating instability and fighting" in his country.
However, he said he was more optimistic than a year ago that behind-the-scenes discussions between his government and the Taliban would prove fruitful.
Mr Karzai said he expected fighting in his country to diminish when foreign troops leave as it will remove a grievance that drove many to take up arms.
He added: "The exit of foreign forces will not bring more violence for them to perpetrate against their own people, but a serious, strong, good reduction in violence will occur."