Standard and Poor's said rescuing institutions after a Yes vote would be "challenging" .
It also warned that many banks could struggle to cope with the SNP's target of declaring independence by March 2016.
Earlier this year Standard and Poor's assessed an independent Scotland's financial standing saying that it faced "significant, but not unsurpassable" challenges.
The latest assessment is an attempt to look at how to rate Scotland's banks, not the country itself, after independence.
The company pointed to figures originally from a Treasury analysis showing the high ratio between Scotland's GDP and the assets of its banks.
Standard and Poor's analyst Giles Edwards said: "In our view, the willingness and ability of a future Scottish government to support its banking system is challenging at this point, not least because the Scottish banking system's assets are currently a high 1254% of Scottish GDP, which compares with an already high 492% of GDP for the UK, and 880% for Iceland in 2007 just before the (country's) banking system collapsed."
The No campaign described the assessment as "sobering".
A Scottish Government spokes-man said ongoing international reforms were designed to ensure no country should have to bail out its banks again.
The Standard and Poor's briefing note also said it was uncertain whether all Scottish banks would continue to be registered north of the border in the event of a Yes vote.
It warned that major banks, many of which are already embroiled in large-scale restruct-uring processes, could find the SNP's aim to declare independence by early 2016 "challenging".
"A swift negotiation on issues such as monetary policy and regulation would help to reduce uncertainty for banks and allow them time to make the necessary changes," it adds.
But the ratings agency stopped short of saying how it would rate an independent Scotland or its banks. Standard and Poor's also said that it took no position in the independence debate.
A spokesman for Better Together said: "This sobering assessment demonstrates that Scottish savings and Scottish mortgages are more secure as part of the UK.
"We saw in 2008 that the broad shoulders of the United Kingdom were able to save Scottish banks like RBS and prevent the type of situation that we saw happen in other small countries happening here."
A Scottish Government spokesman: "Scotland has all the attributes needed to be an economically successful independent country, which is why another recent Standard & Poor's report concluded that 'even without North Sea output', an independent Scotland would qualify for their 'highest economic assessment', and compared Scotland's economy to that of AAA-listed nations.
"All of the points raised in this latest report have been addressed in the detailed macroeconomic framework put forward by the Fiscal Commission Working Group, and accepted by the Scottish Government. As part of a formal monetary union banks would be assessed against consistent regulatory standards, and there would be harmonised deposit protection."