Plans to replace MI5, MI6 and GCHQ, the UK's secret communications base, with a security agency are unaffordable and unrealistic, warned a briefing from the Royal United Services Institute (RUSI).
Instead, said analysts from the defence and security think tank, an independent Scotland should follow Denmark's example and put the police in charge of a limited intelligence-gathering operation.
They concluded: "An independent Scotland is unlikely to face the severity of threats faced by the UK. Given this more benign threat picture, the creation of a Scottish Security and Intelligence Agency seems unnecessary, with more promising avenues including developing and expanding an intelligence division within Police Scotland."
The assessment provoked angry clashes between campaigners in the referendum battle.
Scots Tory leader Ruth Davidson claimed RUSI's report proved Scotland was safer within the UK.
But Allan Burnett, a former senior police officer and Yes Scotland spokesman, insisted a new intelligence service would be a trusted ally of the UK and other countries.
The briefing, whose lead author Charlie Edwards is RUSI's director of national security studies, said the feasibility of the SNP's plans, set out in its White Paper independence blueprint, was "problematic" and "raises serious concerns for Scottish national security". It says the estimated £206 million annual running costs - based on the country's share of UK intelligence spending - was "entirely meaningless" as it was unrelated to an independent Scotland's security needs.
Ignoring set-up costs - particularly the expense of replicating the GCHQ listening centre - "seems a fundamental flaw" in the SNP plans, it added. The report said it was questionable whether a service "of any quality" could be established in time to protect the country from day one of independence, as promised by the SNP.
More seriously, it said the Scottish Government should "not assume co-operation" from the UK on the highly sensitive issue of intelligence-sharing.
It warned an independent Scotland would also not gain automatic entry into the "Five Eyes Agreement", the long-established security pact comprising the UK, US, Canada, Australia and New Zealand.
Raising further doubts over the plans, it added: "An independent Scotland's potentially anti-nuclear policy would likely cause ructions between the Holyrood and Westminster governments."
But it said the danger of Scotland becoming a weak link in the battle against terrorism, organised crime and cyber-espionage would encourage the UK to co-operate with an independent Scotland.
The report said an independent Scotland "could have a first-class security service", but concluded that "economic, diplomatic and technical realities" would dictate a different course.
It recommended "a more modest" model based on Denmark's police-run security service. The service would be run by a deputy chief constable in charge of a new division of Police Scotland and employ 700 to 800 people, the report added.
Ms Davidson said: "The experts at RUSI have confirmed what we already know: that Scotland is safer as part of the UK. Each of the four home nations would be less secure if there was to be an independent Scotland. The safety and security of a nation is the first responsibility of a government, but Alex Salmond is willing to risk that security for his obsession with independence."
Speaking on behalf of Yes Scotland, Mr Burnett, a retired police head of counter-terrorism, said: "An excellent Scottish intelligence organisation could be developed in an independent country. Based upon my experience, I simply do not accept these criticisms of Scotland's abilities to have effective security arrangements.
"Our friends, including those south of the Border, will want Scotland as allies as much as we want them. Our Scottish intelligence service will be welcomed as a professional, trusted ally."