Spending cuts would be required to achieve the Scottish Government's goal to establish a Norway-style oil savings fund, Glasgow University's Centre for Public Policy for Regions (CPPR) said.
Scotland could cut its defence and foreign affairs budget to a level similar to other small independent countries to bridge the gap, the academics said in a report.
The CCPR said: "Oil-related tax revenues would fail to fill the gap left from the loss of Barnett-related UK funding in an independent Scotland.
"However, such an outcome could be overturned, and possibly a small amount could be available to invest in a savings fund or to reduce debt levels, should budget savings be made in public services currently reserved to the UK government, such as defence."
It added: "In order for there to be no net loss to Scottish public finances in moving from the current Barnett arrangement there would need to be an increase in production levels, an increase in prices, or both.
"Such a shift upwards is more likely to occur as a result of a higher oil prices than the production rise needed, as price movements are likely to remain highly erratic.
"However, there is no predictable pattern in the movements of the oil price over time, so one cannot say that such a shift up is more likely than a shift down that would further worsen Scotland's public finances.
"Previous Scottish Government scenarios for future oil revenues have forecast significant short term 'surpluses', amounts surpassing £7 billion. However, these scenarios have proven to be optimistic."
A Norway-style oil fund is "a worthy ambition" and "would be possible if spending was cut", the CCPR said in the paper, entitled Reflections on a post-independence fiscal landscape, including the potential for the development of a savings fund.
It added: "Savings to the currently reserved spending areas such as defence and foreign affairs might be possible, specifically if they were to be in line with the spend per head experienced in similar, small, independent economies.
"However, even if this were manageable, the annual input into such a fund is likely to be relatively small and nothing like the scale seen in Norway."
Alistair Darling, chairman of the pro-UK campaign Better Together, said: "The experts are clear that even with all the money we get from North Sea oil, an independent Scotland would still be in deficit.
"SNP ministers would have you believe that everything will be all right on the night, but the experts are clear that a separate Scotland would have difficult choices to make.
"There would need to be big tax rises, spending cuts impacting the most vulnerable or even more borrowing. Alex Salmond needs to tell us the truth about the cost of independence.
Labour finance spokesman Iain Gray said: "This confirms the inconvenient truth that the SNP dare not admit in public - an independent Scotland would be entirely reliant on the volatility of oil and would mean serious instability in the public finances.
"This report also confirms that an oil fund would have to be paid for by cuts to public spending."
Liberal Democrat leader Willie Rennie said: "Nationalists' independence plans would mean increased taxes, cuts to public spending or more borrowing.
"People don't want to stake their future on a mystery prize. The SNP need to publish the sums behind their independence plans."
A Scottish Government spokeswoman said: "The report confirms that Scotland is in a relatively stronger fiscal position with higher tax receipts per person than the rest of the UK as a whole, and the CPPR report confirms that we are now seeing record levels of investment in the North Sea.
"To ensure the bonus from oil and gas is managed effectively and the benefits are spread across future generations, the Fiscal Commission Working Group has proposed, upon independence, establishing a short-term stabilisation fund and long-term savings fund.
"Had Scotland used its oil wealth to establish an oil fund in 1980, by 2011/12 Scotland could have accumulated financial assets of between £82 billion and £116 billion - equivalent to between 55% and 78% of GDP.
"We welcome the CPPR's conclusion that the economic rationale for an oil savings fund is clear and a 'worthy ambition'. Only independence provides Scotland with the opportunity to make the significant improvements to the tax system which the IFS have highlighted in their report."
A spokesman for Yes Scotland said: "Scotland has got what it takes because without a drop of oil we are 99% of UK GDP per head. With it, we are 117%.
"Tax take per head in Scotland is £1,700 higher than the UK average -something that Mr Darling and the No campaign conveniently forget as part of their overall campaign of misinformation and scare stories. The truth is that Scotland is financially stronger than the UK - a strength that has been consistent for each of the past 30 years.
"Nearly all countries with major oil and gas reserves have created oil funds. But rather than invest in a fund, successive Westminster governments have squandered the wealth from the North Sea. So it makes absolute sense that an independent Scottish government should establish an oil fund as soon as fiscal conditions allow.
"Only a Yes vote next September will give us the opportunity to use our vast wealth and natural resources in a way that benefits all."