Mr Cameron will now call a vote in the House of Commons on Monday on Conservative proposals for a Royal Charter to underpin the new system.
Both Labour and Liberal Democrats voiced dismay at the break-down of discussions designed to establish cross-party consensus on the sensitive proposals in the Leveson report into phone-hacking.
It is unclear whether Deputy Prime Minister Nick Clegg will tell his MPs to vote with Labour to defeat the Tory proposals.
In a hastily-arranged press conference in 10 Downing Street, Mr Cameron announced that the cross-party talks had "concluded without agreement" after a conference call between himself, Mr Clegg and Labour leader Ed Miliband this morning.
He said that the proposals for an independent body, established by Royal Charter, to oversee the system of press self-regulation would provide "the toughest regulation of the press that this country has ever seen".
Newspapers would refuse to sign up to a new system which is underpinned by statute, as recommended by Lord Justice Leveson and advocated by Labour, Lib Dems and the Hacked Off campaign group, he warned.
"The route I have set out is the fastest possible way to deliver the strong self-regulation body that Leveson proposed that can put in place million-pound fines, prominent apologises and get justice for victims in this country," said the Prime Minister.
"The deal is there to be done, it is the fastest way to get proper justice for victims."
A senior Labour source said: "The Prime Minister's decision is very disappointing.
"We still hope for an agreement. We still believe there can be an agreement. We urge the Prime Minister to reflect on his actions."
And a senior Lib Dem source said that Mr Cameron had made the decision to pull the plug on cross-party talks "unilaterally" and the Liberal Democrats were now considering their next step.
"We were very surprised and disappointed," said the source. "We thought we were making real progress and inching towards a deal, but the Prime Minister has unilaterally decided to pull the plug on cross-party talks.
"We are still prepared to work with politicians of all parties, including the Conservatives, who want to work with others to implement Leveson."
Asked whether Lib Dems would vote with Labour against the Royal Charter proposals on Monday, the source said: "We are going to have to talk about it and see what we do. Nothing has been agreed in government."
Mr Cameron said the treatment by the press of individuals such as the families of missing girl Madeleine McCann and murdered teenager Milly Dowler was "absolutely despicable" and that he wanted a new system of press regulation that would prevent such acts happening again, while still protecting the freedom of the media.
But he said that - as he warned in his initial response to the Leveson Report in November - a press regulation system backed by statute would "cross the Rubicon" towards endangering media freedom.
"That is why we propose the Royal Charter approach," he said. "It is a proven way of establishing a public body without the need for legislation as we've seen with, for instance, our universities or the BBC.
"It would deliver, a Royal Charter, what Lord Justice Leveson wanted without the need for detailed legislation.
"The Royal Charter that we've proposed has benefited hugely from hundreds of hours of detailed negotiations with the representatives of victims, with all main political parties and with the press themselves.
"A Royal Charter would ensure the independent self-regulation that Lord Justice Leveson recommended while simultaneously protecting the precious independence and freedom of the press.
"It follows the principles that Lord Justice Leveson set out. It would deliver up-front apologies, million pound fines, a self-regulatory body with independence of appointments and funding, a standards code, an arbitration service free for victims, and a speedy complaint handling mechanism.
"All of the fundamental principles set out by the Leveson Inquiry are met by the Royal Charter that we propose and all have been accepted by the industry over the last three months. In fact, in a number of areas, the media have accepted additional measures that go beyond Lord Justice Leveson's recommendations - these include a dedicated fund for investigations, making publishers accountable for all material including photos, and a whistle-blowing hotline.
"As a result of all of this, we have a workable system ready to go."
Mr Cameron pointed out that Leveson said any self-regulation system for the press should be voluntary.
"There's no point in producing a system that the press won't take part in," he said.
"As Prime Minister I wouldn't be fulfilling my duty if I came up with something knowing that it wouldn't work.
"I believe that what we have on the table is a system that will deliver public confidence and justice for the victims.
"It's a system that would introduce the toughest press regulation this country has seen and a system that will defend press freedom in our country."
Mr Cameron said he would publish the Royal Charter, and would then use amendments to the Crime and Courts Bill to table "the minimal legislative clauses needed to put in place a system of exemplary damages for any newspaper that chooses to remain outside the regulatory system".
He added: "This delivers what Lord Leveson proposed - namely, a strong incentive for newspapers to take part in the new regulatory system.
"I would urge all members on all sides of the House to support this approach."
Mr Cameron said he wanted to unblock legislation which he said had been "hijacked" by amendments seeking to bring in clauses about press regulation.
He said: "What I can't do, and what I think would be wrong for the country, is to just sit back and watch while bill after bill going through Parliament is sort of hijacked by a set of clauses about press regulation.
"We saw that on the Defamation Bill, we saw that on the bill to introduce the very important Green Investment Bank, and if we did nothing we would see exactly the same thing on the Crime and Courts Bill, which establishes the National Crime Agency, an absolutely key reform for the safety and security of our country.
"So it is better to bring this to a head, to say to the House of Commons: 'Here is my approach, back it and we will get what we need in terms of press regulation'.
"But if other parties want to come forward with a different approach that can be voted on on Monday night and the House of Commons says that is the case, then our country will be sovereign and will decide.
"I think if everyone opts for a full-on Leveson solution, I don't think it would work, I don't think it is the right step forward for our country.
"I think it would be a mistake, but what I can't do with effectively a hung parliament is say that we will negotiate forever, never reaching a conclusion, and having bill after bill wrecked by these tactics. We need to bring this to a head, make a decision and, in my view, put in place something that can work, and work quickly."
He said the Government's clauses on exemplary damages would give the press "strong incentives" to sign up to the proposals. But he said the Government's legislation did not mention the roles and functions of the regulator and it does not legislate about the detail about press regulation.
He told a press conference he would appeal beyond his party for support.
Mr Cameron said other parties would have the opportunity to table their own amendments to the Bill ahead of Monday's vote.
"To put it simply, they can back my amendments and support this Royal Charter and secure a workable new system that delivers the principles of Leveson's recommendations or they can grandstand and end up with a system that I believe will not work," he said.
"The only way we can help victims is through a system that actually works in practice."
Mr Cameron said a "full legislative response" would be wrong on grounds of "necessity, practicality and fundamental principle".
Statutory regulation is not necessary to achieve the Leveson principles, he said.
And he said it would not be practical because "no system of self-regulation can work unless those who are being regulated participate in it".
He added: "Most important of all, I believe that detailed legislation is fundamentally wrong in principle. It is wrong to cross that Rubicon by writing key elements of press regulation into the law of the land.
"It is wrong to create a vehicle whereby politicians could in future impose regulations and obligations on the free press.
"And it is wrong to run the risk of infringing free speech and a free press in this way."
Mr Cameron cited Sir Winston Churchill's dictum that the free press is "the unsleeping guardian of every other right that free people prize (and) the most dangerous foe of tyranny".
And he added: "He was right. We should always defend that principle, and I will."
