The SNP insists Scots will not lose the currency if they vote Yes in September's referendum.
But First Minister Alex Salmond is under pressure to set out a Plan B after Chancellor George Osborne ruled out a sterling zone.
Currencies Direct says continuing to use a currency without a formal union, as is the case with Panama and the US, would not be a viable option.
It would leave an independent Scotland without the economic levers to fully support its booming financial sector, it warns.
But the best alternative - an independent Scotland's own currency - carries its own risks and problems, it claims.
The First Minister is "caught between a rock and a hard place" on the issue, it adds.
The company, which helps small businesses involved in export or import with currency conversion, urged the SNP leader to set out his preferred roadmap on a currency, to ease market uncertainties.
Earlier this month, all three major parties at Westminster ruled out a currency union between an independent Scotland and the remaining UK.
The SNP insist that this stance is a bluff and say they will continue to argue for a sterling zone.
However, yesterday Mr Salmond remarked that there was "not a question" about Scotland keeping the pound, a comment widely interpreted as hinting at his preferred Plan B.
Alistair Cotton, Currencies Direct head of corporate dealing, said: "Setting politics aside, without a doubt the best long-term option is setting up a Scottish currency with a floating exchange rate.
"It gives Scotland full control over its finances and allows for flexibility as economic imbalances build - particularly useful if oil receipts are to form a key part of the Scottish economy.
"But it will impose enormous short-term costs and logistical challenges, which would be circumvented by joining either sterling or the euro.
"As the former appears to be increasingly unlikely and the latter is unpopular with voters, Mr Salmond is caught between a rock and a hard place."
Currencies Direct believes there are two routes open to an independent Scotland - either be part of a formal union with sterling or the euro, or create its own currency.
But setting up its own currency would involve an "arduous" set of hurdles, including the creation of a new central bank, regulator and payments system, which it warns could potentially take years of planning to get right.
The company also says that an independent Scotland would have to agree to honour a "sizeable" amount of pre-independence UK Government bonds.
This would be necessary to create market confidence, it warns.
The new government of an independent Scotland would also have to change all existing sterling contracts into the new currency, including everything from Scottish mobile phone bills to private/public finance deals that pay for building Scottish hospitals.
Last night a spokesman for Better Together said: "The nationalists are in utter chaos over currency. Alex Salmond's Panama Plan would be an absolute disaster for Scotland, leaving us without anyone standing behind our banks and resulting in sky-high interests rates for our mortgages and loans. The only way to keep the strength and security of the pound is to vote to stay part of the UK."
A Yes Scotland spokesman said: "People in Scotland can be assured that we will be using the pound after a Yes vote. The pound belongs as much to Scotland as it does to the rest of the UK."