Scotland would face "significant resourcing, capability and legislative hurdles" to replace the security currently provided by MI5, MI6 and GCHQ as part of the UK, a report by the Royal United Services Institute (RUSI) said.
But the terror threats to an independent Scotland would be "less severe" meaning a full security service may be unnecessary, the report by RUSI director of national security and resilience Charlie Edwards said.
Scotland should consider an expanded intelligence division within Police Scotland rather than the £206 million security and intelligence agency envisaged in the Scottish Government's White Paper on independence.
The SNP's price tag is "entirely meaningless" as it does not take account of the set-up and training costs required, and it is "questionable" whether it could be established in time for the proposed independence day in March 2016, according to RUSI.
While intelligence sharing with the UK and its partners should not be assumed, Westminster would have a "strong incentive" to co-operate with Scotland to prevent a security gap on its northern border, RUSI said.
The report states: "An independent Scotland is unlikely to face the severity of threats currently faced by the UK.
"Given this more benign threat picture, the creation of a new Scottish security and intelligence agency seems unnecessary, with more promising avenues including developing and expanding an intelligence division within Police Scotland.
"Significant resourcing, capability and legislative hurdles would affect the creation of a new Scottish security and intelligence agency. This would take years to develop, and create short-term vulnerabilities for Scottish national security.
"A Scottish security and intelligence agency would not have automatic access to 'Five-Eyes' intelligence, but would need to negotiate some limited access. This could severely constrain a new agency's investigations and its ability to protect an independent Scotland.
"This also creates a potential vulnerability for wider UK national security. A weak Scottish security and intelligence capability could make Scotland an attractive environment for hostile intelligence organisations, and provide a route into the UK. As a result, the Westminster Government would have a strong incentive to co-operate in attempts to mitigate this problem.
"Were co-operation to be ineffective, the perception of Scotland as a vulnerability to the UK could exacerbate tensions between the Holyrood and Westminster governments over security issues."
It added: "Developing an intelligence division within Police Scotland is the most promising avenue to explore.
"This would potentially cost less than the amount the White Paper currently allocates to the agency, although we accept that even this scenario would have cost implications.
"The merger of police forces was not without problems and it is worth raising them again in light of the plan to create a Scottish Security and Intelligence Agency. The move to a single police service was hampered by poor information, a lack of clarity about roles, and difficult relationships between the Government, Police Scotland and the Scottish Police Authority.
"There have been significant changes to governance arrangements, and some important strategies and plans are still under development.
"This major reform of the Scottish police aims to save £1.1 billion by 2026, but, according to Audit Scotland, it is not clear how these savings will be achieved.
"On this basis we question whether the Scottish Government can actually afford to create an entirely new agency. Significant resourcing, capability and legislative hurdles in the creation of a Scottish Security and Intelligence Agency remain.
"An independent Scotland could have a first-class security and intelligence capability, but the political arguments for an intelligence agency are no match for the economic, diplomatic and technical realities of developing a new intelligence division in Police Scotland."
Scottish Conservative leader Ruth Davidson said: "The experts at RUSI have confirmed what we already know - that Scotland is safer as part of the UK.
"Each of the four home nations would be less secure if there was to be an independent Scotland.
"The safety and security of a nation is the first responsibility of a government, but Alex Salmond is willing to risk that security for his obsession with independence."