Theresa May said terrorist and criminal threats may not diminish in an independent Scotland but the ability to detect and prevent them might be reduced.
Her comments came as the UK Government launched its seventh analysis paper on Scottish independence, focusing on security.
It concludes that threats to Scotland from organised crime gangs, cyber criminals and global terrorism "are best confronted with Scotland inside the UK".
However, Scottish Justice Secretary Kenny MacAskill said: "These claims are wrong - not least because Scotland is already an independent jurisdiction when it comes to policing and justice issues, and current cross-border cooperation shows how well that can work to combat terrorism and other threats.
"An independent Scotland will have first rate security arrangements to counter any threats we may face. And we will continue to work in very close collaboration with the rest of the UK and international partners on security and intelligence matters, which is in everyone's interests."
SNP MSP Christine Grahame, who chairs Holyrood's Justice Committee, said: "This is Project Fear at its worst.
"Trying to politicise issues of security and anti-terrorism in this way is the height of irresponsibility.
"This paper completely ignores the fact that Scotland is already an independent jurisdiction for justice and policing, and as such we already have lots of cross-border cooperation on counter-terrorism and other issues which will continue in an independent Scotland.
"It frankly beggars belief that Theresa May feels able to lecture anyone on arresting people in other countries after she was forced into an embarrassing U-turn on the European Arrest Warrant earlier this year.
"While Theresa May's own record on security is one of cutting police officers by over 11,000 and undermining policy pay and conditions south of the border, the Scottish Government has a strong track record on standing by our police service.
"In Scotland we have delivered more than 1,000 additional police officers while protecting them from the deeply damaging Winsor changes to pay and conditions.
"The SNP has a record of delivering when it comes to policing and security and we will take no lessons from someone who has done so much to undermine this vital area south of the border."
Blair Jenkins, chief executive of the Yes Scotland pro-independence campaign, said: "This amounts to yet another Project Fear fiction from a Tory minister on a flying visit to talk Scotland down.
"Security is very much a matter of international co-operation and we look forward to Scotland's security services playing their part. As our nearest neighbours, we can expect a particularly close working relationship with the security services of the UK, to our mutual best interests.
"That makes sense for Scotland and for the rest of the UK and Theresa May knows it."
He cited the weekend comments of former Scottish police counter-terrorism co-ordinator Allan Burnett, who envisaged an independent Scottish intelligence and security service of "huge ability, energy, integrity and innovation".
Mr Burnett said: "I believe an independent Scotland would face less of a threat from terrorism for a number of reasons. We would not add to international tension by taking part in illegal wars and, as a nuclear-free state, potential terrorist targets would be removed from our country."
Mrs May launched the paper in Edinburgh today and rejected nationalist arguments that Scotland may face a diminished threat from terrorism by shunning nuclear weapons and "illegal wars" such as the Iraq conflict.
She said some terrorists have "an issue with the way of life in the west" rather than any specific grievances about foreign policy.
"I don't think it is possible to guarantee that the threat would diminish with a separate Scotland," she said.
"There are a number of issues that give rise to the threat that is currently facing the UK.
"There are those for whom it is not just country-specific because of foreign policy, or actions that have taken place, but there is an issue about the West and the way of life in the West.
"But what would change would be the scale of capability that Scotland would have access to."
The UK's current intelligence and security network could not be easily divided and resources such as the £860 million UK cyber security programme "would not be available to Scotland", she said.
Mrs May added: "Separate bits could not be hived off. This is one of the challenges, one of the hard issues that the SNP need to address.
"We took a decision when we came into government in 2010 that we were going to put significant resource into improving cyber security of £860 million over five years. That would not be available to Scotland as a separate state."
The Scottish Government argues that assets, including defence and security assets, should be divided fairly between Scotland and the remainder of the UK, but Mrs May said the SNP must articulate how they would deploy their security assets.
"The question in relation to funding and budgets is for the Scottish Government to actually answer the question of what they are looking to produce," she said.
"What would their security service be like? This is an area where they have been singularly unclear and hazy up to now."
The UK Government would not prevent UK security personnel from joining an independent Scottish security service, she said.
"Obviously people do move away from the existing security organisations and work in other spheres, so it is for individuals to decide if they wished to be recruited into another organisation.
"It isn't the case that we would prevent people from ever leaving the security services and working elsewhere."
In a written ministerial statement presenting the paper to Parliament, Mrs May said independence would "impact negatively" on Scotland's ability to pursue justice across borders and defend itself from foreign attacks.
"The UK's security and intelligence agencies protect against terrorism and espionage, enable the collection of secret foreign intelligence, and make possible intelligence-gathering through the monitoring of communications," she said.
"An independent Scottish state would lose automatic access to UK intelligence, impacting on its ability to counter espionage and hostile foreign intelligence activity as well as terrorism.
"We are better protected through enhanced justice and policing capabilities and international partnerships to tackle serious and organised crime."
Scottish people are a proven "target for attack" by terrorists, gangs and cyber-criminals, the paper warns.
These threats "will persist in the future" but independence "would result in an increased risk to the Scottish people", it said.
It cites the Glasgow Airport bombing in 2007, the conviction of a man in Scotland for funding an Islamist attack in Stockholm in 2010, and the Scots caught up in the In Amenas gas plant attack in Algeria in January this year.
The UK is currently under "substantial" risk of terrorist attack, meaning an attack "is a strong possibility and may occur without warning" at any time.
The paper stated: "Terrorists, organised criminals and those who conduct cyber attacks have demonstrated that they consider Scottish interests, people, and infrastructure targets for attack.
"Terrorist groups have demonstrated that they are unconcerned by differences in nationality amongst Western countries; Scottish people suffered directly at the hands of terrorists in Glasgow in June 2007 and the In Amenas attack in Algeria in January 2013.
"Organised criminals have and will continue to target Scottish people and their interests because the perceived benefits of such crimes remain. The perpetrators of cyber attacks, including cyber espionage, have shown little regard for international boundaries. These threats will persist in the future.
"The creation of an independent Scottish state will, however, without proper planning and investment, lead to a reduction in the capability of its government to protect Scottish interests, infrastructure and people from these threats. A reduced capability to tackle a persistent threat would result in an increased risk to the Scottish people."
Former home secretary Lord Reid, a Scottish Labour politician, said: "There is no doubt that a separate Scotland would be less secure than it is at present and less well-placed to counter many of today's growing security threats.
"Scotland could, of course, separate from the UK, but it cannot separate itself from the substantial and growing threats of international crime, terrorism or cyber attack.
"Moreover, a separate Scotland would need to massively reduce security spending and operations from that of the UK, but the threat would not be commensurately reduced.
"We would therefore be left with the same level of threat but with a much-reduced capacity to respond. A separate Scotland would inevitably be in a worse place in terms of scale, effectiveness, intelligence and influence than it is as an integral part of the UK."