The 81% taxpayer-owned lender avoided a threatened full split into a "good" and "bad" bank as a Government-commissioned report concluded that would do more harm than good.
Instead RBS will create an internal bad bank of problem assets to be run off over the next three years, avoiding the "effort, risk and expense" of the fully nationalised alternative.
But Chancellor George Osborne admitted the bank, which plunged to £634 million third-quarter losses, is "unlikely" to be sold back to the private sector before the 2015 general election.
RBS shares slumped almost 6% as the results missed City forecasts and the NatWest owner raised the spectre of up to an extra £4.5 billion in bad debt writedowns in the final three months of the year as future loan losses are dragged forward.
New chief executive Ross McEwan has also started a deep review which will report back in February and will see heavy cost-cutting and a sharper focus on business lending. It will also speed up the sale of its Citizens US banking subsidiary, with a partial flotation next year.
The "bad bank" report was published alongside two more lengthy reports, one detailing RBS's performance between July and September, and another criticising the bank's lending to small businesses.
The scathing report by former deputy governor of the Bank of England, Sir Andrew Large, found a host of problems in the way RBS treats small and medium-sized enterprises (SMEs) - including long delays on approving loans and poor treatment of struggling borrowers.
Mr McEwan said the bank accepted the report and will address it in its review, which aims to "create a bank that can reward the faith of UK taxpayers and all our investors".
He said: "We are hugely indebted to the public of the UK for putting their hands into their pockets and saving this great institution."
RBS, which needed a £45 billion taxpayer bailout in 2008, said the bad bank will contain about £9 billion of assets from Ulster Bank, as well as problem commercial property loans.
The lender said it has already done the "vast bulk" of repairing its balance sheet, slashing "non-core" assets from £258 billion five years ago.
But it said the new plan will boost its financial strength by releasing capital, as well as liberating management.
The future of Ulster Bank, a major lender in Northern Ireland and the Republic of Ireland, will be decided in February's review. Mr McEwan refused to say whether its cost-cutting plans will entail heavy job losses.
It wants to reduce its ratio of cost to income from 65% to the mid-50s.
Trade union Unite warned against further redundancies and said the bank has already "cut staff numbers to the bone" after 30,000 job cuts.
Unite national officer Dominic Hook said: "RBS must make a commitment to end the constant cuts which threaten the service to customers that the bank is staking its reputation on."
The review by City firms Rothschild and Blackrock found that an external bad bank would be unlikely to speed up a return of RBS to the private sector, compared with an internal bad bank.
Its outcome was welcomed by the Bank of England, which said it should "create a more resilient institution that is better able to support the real economy without any expectation of further Government support".
It appeared to mark a change of tack from the central bank, after former Bank of England governor Sir Mervyn King backed a full split of the lender. Andrew Tyrie, chair of the Parliamentary Commission on Banking Standards, had also called for a break-up.
Mr Tyrie said: "We are now in a better place - we have agreement that vigorous action is required to sort out RBS' SME lending strategy."
The Treasury also said taxpayer support for the lender has been reduced by another £8 billion after the so-called Contingent Capital Facility was removed a year early.
It is also in talks with the European Commission on speeding up a return to dividend payouts - by removing the prohibitive Dividend Access Share.
Mr Osborne said while ring-fencing problem assets should make it easier to "get our money back", a sale of RBS is "still some way off".
He told BBC Radio 4's Today programme: "I think, quite frankly, it is unlikely before the general election."
RBS's pre-tax losses of £634 million compared with £1.37 billion losses a year earlier, and were hurt by one-off items and an additional charge of £250 million to cover mis-selling of payment protection insurance (PPI).
RBS also booked £99 million of charges for unspecified "regulatory and legal actions". It has reportedly suspended two traders in its foreign exchange arm as regulators clamp down on manipulation of currency markets.
Mr McEwan refused to comment on the case but said it will "come down very severely on anyone we discover has been breaking the rules".
Operating profits more than halved to £438 million from £909 million.
Investec analyst Ian Gordon said: "Any relief at the avoidance of a full break-up is tempered by significant shareholder value destruction in measures announced today."
Mr Tyrie said the Treasury Select Committee he chairs would examine whether the move was another "missed opportunity" to take decisive action to turn round RBS.
He told BBC Radio 4's World at One: "We've had several opportunities to put this right. We could have put this right in 2008 and had a full split then. The second opportunity came and went in 2010.
"Both of those, there were difficult decisions to take because of the politics involved. I quite understand, particularly in the emergency situation of 2008 why a small residual holding was left which complicates matters a lot and why it wasn't fully nationalised.
"Now we've had this third opportunity. We are going to analyse very carefully whether this is a missed opportunity or whether the Government's got it right this time, and I hope they have because that's what the economy needs."
Labour former chancellor Alistair Darling, who was responsible for the state intervention, backed the Government's decision and said it was not possible for him to split the bank at the height of the banking crisis.
He told the programme: "In the case of RBS we had a matter of hours to stop it collapsing. The restructuring option simply wasn't available there.
"The second thing is RBS at that time was probably one of the biggest, if not the biggest bank in the world and it took a long time in order to establish what was good in the bank and what was bad."
He said a lot of the bad loans would have to be written down, particularly the Irish property assets which would need to be sold at a loss.
"You need to remember what these things actually are. They are basically lending on, in many cases, Irish commercial property which frankly is never going to recover," he said.
"You are not going to get your money back on that and I'm afraid that's just a consequence of some of the foolish decisions taken by, not just RBS but other banks at the time, because they have all had to engage in quite extravagant write-offs."