A separate Scottish banking market would create additional costs for banks on both sides of the border, according to an international ratings agency.
An independent Scotland's choice of monetary regime will be crucial in assessing how credit worthy its banks would be, a report by Standard & Poor's said.
The Scottish Government wants to share the pound in a formal currency union with the Bank of England, but all major UK parties have pointedly ruled this out.
If Westminster's threat is carried through, Scotland would need to guarantee banks with assets over 10 times the size of its GDP, according to S&P.
The absence of an effective deposit guarantee scheme could see a flight of Scotland's banks across the border, it said.
"The willingness and ability of a future Scottish government to support its banking system appears challenging to us at this point, not least because system assets could be over 1,000% of Scottish GDP," S&P said.
"A new banking market would mean additional costs and risks.
"For Scotland and remaining UK-domiciled banks alike, the creation of a new banking market would likely create potential additional costs, and some associated operational risks, in reconfiguring operating models (systems, staff and legal entity structures) and products.
"These would likely be further accentuated if Scotland did not use the British pound sterling."
The Bank of England "would likely remain available as lender of last resort" in a formal currency union, while Scotland would have access to the European Central Bank if it joined the euro.
But the creation of a new Scottish currency or the unilateral use of the pound would be more problematic, S&P said.
"If Scotland adopted its own currency, we assume it would set up a central bank to operate monetary policy and presumably provide the lender of last resort function to domestic banks.
"However, in a stress scenario, internationally active Scottish banks would likely have substantial liquidity needs in sterling, and this may be unavailable from a Scottish central bank unless it had high foreign currency reserves or a negotiated swap line with the Bank of England.
"By contrast, if Scotland uses sterling unofficially, then it would likely have no central bank. Scotland-domiciled banks would very likely have a right of access to Bank of England liquidity through any remaining UK subsidiaries, although the right of access of their UK branches could be more constrained.
"In our view, this could be a clear point of weakness for a Scottish banking system compared with most banking systems worldwide."
It continued: "If it became an EU member, Scotland would be required under EU law to establish and maintain an effective deposit insurance scheme.
"In our view, credible deposit insurance arrangements would be essential in helping to underpin the quality and price-effectiveness of Scottish banks' deposit bases, and, in a stress scenario, to mitigate the risk of deposit flight over the border."
Banks could need "considerable time" to adjust to an independent Scotland and it could be "a challenge" to complete the necessary reforms by the proposed independence day of March 2016, S&P said.