In its latest analysis of the consequences of independence, focusing on borders and citizenship, it says it is likely that Scots would be able to hold dual Scottish and British nationality.
The next generation of Scots would also be British, the paper suggests, as children inherit citizenship under present UK rules.
Their children would not be British, as British citizens outside the UK cannot pass on their citizenship more than one generation.
As British citizens, Scots would keep their existing passports and continue to qualify for UK consular help abroad.
The Home Office document, the 10th in the Government's Scotland Analysis series, warns: "This could raise complex and difficult issues for individuals and families."
Government sources stressed existing citizenship rules could be changed by a future UK Government and there were no guarantees about British citizenship.
Nevertheless, the prospect of Scots remaining British, set out by the Home Office, was welcomed by the Scottish Government as a common sense move.
The Home Office paper said: "The UK has ... been tolerant of plural nationalities, and it is likely there would be no barriers to holding both British and independent Scottish citizenships."
While appearing to offer British citizenship to people living in an independent Scotland, however, the UK Government questioned the SNP's plans to offer Scottish nationality as widely as possible.
It warned plans to follow Ireland's example, and allow anyone with a Scots grandparent to become a Scottish citizen, could strain relations with the EU.
The report said the number of new citizens could equal or exceed Scotland's current population and citizenship could be used to move to other EU countries.
The document warned the SNP's plans for a steep increase in immigration in the event of a Yes vote could prevent Scotland from joining the Common Travel Area between the UK and Ireland.
The CTA, which allows borderless travel between the UK and Ireland, relied on both countries pursuing similar immigration policies, it said.
Scottish Secretary Alistair Carmichael said plans to ease restrictions on immigration while seeking to join the CTA were contradictory.
He warned: "I think it's pretty well accepted that you can have either of these things but you cannot have them both."
The document said an independent Scotland would be required under EU rules to put up checkpoints along the Border with England if it failed to negotiate an opt-out from the Schengen Area, the much bigger free travel zone covering most of Europe apart from the UK and Ireland. It also warned that a different immigration policy from the rest of the UK could lead to border posts.
A spokesman for external affairs secretary Fiona Hyslop said: "One thing notable about this paper is that the UK Government now accept there 'would be no barriers' to dual citizenship post-independence.
"That is a welcome climbdown from the scare stories Home Secretary Theresa May was promoting six months ago, and follows the common sense climbdown from the Treasury last week over debt."