Talks chairman Dr Richard Haass, a former US diplomat, said he had not managed to secure consensus on a final set of proposals to deal with flags, disputed parades and the legacy of the Troubles before his end-of-year deadline.
Dr Haass said a working group made up of representatives of the five parties in Stormont's power-sharing executive would now be set up to try and find another way to build on "significant progress" that had been achieved.
Negotiators from Sinn Fein, the largest nationalist party in the Executive, said they were prepared to recommend the proposals to its ruling executive, but unionists would not sign up to the document tonight.
"Yes it would have been nice to come out here tonight and say we have got all five parties completely signed on to the text, we are not there," Dr Haass said.
Dr Haass, who was commissioned by Democratic Unionist First Minister Peter Robinson and Sinn Fein Deputy First Minister Martin McGuinness to chair the six-month process, said he believed there was a prospect that all the parties would either endorse all, or significant parts of his document in the future.
The DUP and Ulster Unionists said they would consult within their parties before making a final judgment on the proposals but both indicated they had major difficulties with elements of the text.
The Social Democratic and Labour Party (SDLP) said it would also be conducting a consultation, but party leader Alasdair McDonnell said he would be recommending a general endorsement of the proposals.
Dr Haass urged Mr Robinson and Mr McGuinness to make the details of the final document public so people could make up their own minds.
He denied the process had been a failure.
"Success should not be measured by what we report to you tonight or what the party leaders report tonight - I would ask you to judge the success in six months, in a year, 18 months, in two years, that would give a much more realistic definition or yardstick of what constitutes success," he said.
"What I believe what we have done is laid down solid enough foundations stones."
Dr Haass and talks vice-chairman Dr Meghan O'Sullivan, a US foreign affairs expert, said their role in any future political process would be limited, but both insisted they were not washing their hands of the process.
Alliance party deputy leader Naomi Long, who along with colleagues was mandated to make a final call on behalf of the party, said she was willing to endorse proposals on the past, but not on flags and parades.
Sinn Fein president Gerry Adams signalled his party's willingness to strike a full deal.
He said the proposals presented by Dr Haass represented a "compromise position" and provided the basis for agreement.
"They aren't perfect, we have had to stretch ourselves to embrace them," he said.
Mr Adams insisted talks could not continue forever and at some point parties had to "call it".
"The paper produced by Dr Haass does in the view of our negotiating team provide the basis for agreement," he said.
He said if there was no progress from this point he would be seeking an urgent meeting with the British and Irish governments to call for the production of a road map towards resolution.
DUP negotiator Jeffrey Donaldson said progress had been made but said a number of difficulties remained.
"We do not have an agreement this evening but we are committed to continuing this work beyond now in dialogue with others to try and resolve the outstandiing issues that need to be addressed," he said.
"We owe that to the people of Northern Ireland, especially to the innocent victims of terrorism who have suffered so much over the decades."
The Haass process was set up in July to deal with what have become three of the primary obstacles to meaningful reconciliation in Northern Ireland.
Tensions over contentious parades regularly erupt into street violence while disputes over the flying of flags - both on public buildings and in loyalist and republican neighbourhoods - continue to be a source of community conflict.
But arguably the most complex issue has been how Northern Ireland deals with the legacy of a 30-year-conflict where opposing sides retain competing narratives of what happened and victims still demand both truth and justice regarding thousands of unsolved murders.
Dr Haass had initially hoped to strike a comprehensive settlement dealing with all three elements in full, but it became clear from the outset of the intensive negotiation phase earlier this month that was going to be highly unlikely.
The last day of negotiations began around 10am yesterday and effectively carried straight through to around 5am this morning.
There was little or no progress made on flags with a proposal to set up a commission to examine the issues over a longer term.
It is understood the document also proposes the replacement of the Government-appointed Parades Commission with another set of structures to adjudicate on contentious marches.
The text also envisaged a new mechanism to oversee dealing with the legacy of the past - with a truth recovery body that would potentially offer limited immunity from prosecution to those who co-operate.
Unionists have indicated concerns with some of the language used and claimed too much focus has been placed on killings perpetrated by state forces.
UUP leader Mike Nesbitt said he had an opinion on the document but would not make it public until his party had the chance to examine the proposals.
"We will have an honest debate and hopefully form a final opinion at the end of that debate," he said.
Mr McDonnell said "bigger and better solutions" were needed on some aspects, but said he would be recommending that his party give a general endorsement to what had been proposed.
While Mrs Long criticised proposals on parades and flags, she said great work had been done in the past.
"We have seen a huge sea change in the level of political agreement which has exceeded public expectation, particularly in delivering for the victims and the reconciliation process," said the East Belfast MP.
Having been given an end-of-year deadline to report, Dr Haass had aimed to strike a deal before Christmas but had to return to the US on Christmas Eve empty-handed after a marathon session of all-night negotiations last week.
Cutting short his seasonal break, he returned to the region on Saturday in a last-ditch bid to secure agreement.
Dr Haass is the president of US think tank the Council on Foreign Relations, based in New York, and was US president George W Bush's special envoy to Northern Ireland from 2001 to 2003.