The National Institute for Economic and Social Research (NIESR) said a go-it-alone Scotland could start life more than £150 billion in the red, a higher figure than previously estimated.
In a report it warned Scots would face a decade of austerity as the nation battled to reduce the debt to a more manageable level.
The think-tank suggested an independent Scotland should have its own currency to give it greater policy flexibility to cut borrowing and should consider using North Sea cash to help pay off the rest of the UK.
The report said Scotland's share of UK national debt would be £153bn, using the EU's preferred measure, in 2016/17, the year the country would become independent if Scots vote Yes.
The debt would amount to 86% of gross domestic product and reducing it to the EU target of 60% would require a decade of austerity similar in severity to that imposed by Chancellor George Osborne since 2010.
The politically neutral NIESR said the task would be made harder as a newly independent Scotland would face higher borrowing costs than the rest of the UK, with interest rates rising by between 0.72% and 1.65%.
The report added: "Such a fiscal tightening would leave an independent Scotland with little room for fiscal manoeuvre in the case of a negative shock, such as a drop in the oil price or a recession.
"That makes it important to have other policies available, such as interest rate or exchange rate adjustment. Among the currency options open ... having its own currency would give it flexibility to respond to shocks."
It also said an "oil for debt swap" - in which the UK would be handed North Sea revenues to pay off a chunk of Scotland's share of the national debt - "would greatly reduce the economic risks".
Previous estimates of Scotland's national debt have been lower as they have been based on past years' figures. The NIESR used Office for Budget Responsibility (OBR) projections for 2016/17.
First Minister Alex Salmond has rejected the idea of Scotland creating its own currency. He insists Scotland would enter a formal sterling zone with the UK, despite Treasury warnings the plan is "unlikely" to be accepted.
Scottish Labour's constitutional spokesman Drew Smith said: "This report highlights again the need for clarity from the SNP on their full currency plans.
"If they are wedded to the idea of keeping the UK pound, without being in the UK, then they need to show how losing control over our monetary unit would be of benefit to Scotland."
l Alex Salmond's former head of policy has criticised his strategy in a blog. Alex Bell, who left the Government in the summer, claimed the referendum campaign was based on "old songs and tired policies".