The banks are understood to have been due to begin testing systems on November 1, ahead of the changeover of 1.8 million customers when the agreement was due to complete in the middle of next year.
The sale was ordered by the European Commission as one of the conditions of RBS receiving a £45 billion bailout at the height of the financial crisis.
Santander is thought to have pulled out following difficulties in integrating the IT systems of the two banks.
An agreement in August 2010 initially valued the branches at about £1.65bn.
Early predictions of completing the transaction by the end of 2011 proved optimistic and the complex deal has been repeatedly delayed.
The poor economic performance in recent years has also hit the financial returns at the branches. Reports earlier this year suggested the changes in profitability had led Santander to begin negotiating to pay a lower price.
It is not known whether the Spanish bank will have to pay a break fee to RBS for calling off the deal.
The branches are predominantly in the north-west of England although there are six NatWest sites in Scotland.
The failure to complete the transaction comes just days after RBS chief executive Stephen Hester told The Herald he was hopeful the turnaround of the bank was on course to be completed by the end of 2013.
Mr Hester faces an anxious few months, with the size of the bank's penalty for the alleged manipulation of Libor, the interbank borrowing rate, likely to be revealed before the end of the year.
RBS will now be forced to try and sell the branches again.
It has until the end of 2013 under the conditions set out by the EU. However, finding a buyer may prove difficult.
Santander was the only bidder the first time around and similar branch sales by competitors have also proved complicated.
Lloyds took several months to finalise a sale of 632 branches to Co-op and had to accept a payment – £350 million up front with prospects for a further £400m – that was lower than initial valuations.
However, in a statement last night Mr Hester said most of the "heavy lifting" to get the branches ready for separation had already been done.
Data for the customers has been separated, meaning any potential new sale is unlikely to take as long as the Santander process.
The collapse of the deal may not be catastrophic to RBS in the long term, but it will make achieving its short-term targets a lot more tricky.
Although Mr Hester has offered no timescales on when RBS may return to private hands, setbacks like the Santander deal are likely to push that date further back.
Contextual targeting label: