The Edinburgh-based bank has been told by a New York judge to defend itself against claims that it committed fraud, and aided and abetted the crime, by inducing it to insure a $1.15 billion (£696m)securities offering backed by the loans.
Bermuda-based Assured Guaranty Municipal Corp is behind bringing the case, which it says surrounds "one of the worst performing residential mortgage-backed security transactions of the last decade".
It is one of a number in the US accusing banks that packaged mortgages into securities prior to the 2008 financial crisis of deceiving investors and insurers about underwriting quality, resulting in losses when market conditions deteriorated.
The claim against RBS surrounds mortgage investment vehicle Soundview Home Loan Trust 2007-WMC1, which held more than 4900 home loans taken over by a wing of the bank.
The Trust was one of many set up that bundled thousands of residential mortgages together. Investors bought shares in each bundle or trust in a process called securitisation. Assured says RBS induced it to insure part of the 2007 securitisation and exposed it to more than $100m of potential losses.
District Judge John Koeltl rejected RBS's request to dismiss Assured's claims of fraud and aiding and abetting fraud and said the complaint was sufficient to raise "a strong inference that RBS had a specific motive and intent to deceive the plaintiff".
Investors lost money when many of the mortgages defaulted, causing the security certificates to drop from AAA-rated to junk status, according to Assured, which said it insured interest and principal payments on $291m (£176m) of the Soundview securities certificates.
The issued certificates represented varying interests in the cash flow from interest and principal payments on the mortgages.
Assured complained it would not have been involved had it known the original loans originally arranged by WMC Mortgage Corp did not conform with prudent underwriting standards that took into account borrowers' ability to repay and that mortgaged properties provided adequate security in the event borrowers default. It said the mortgages experienced "extraordinarily high rates of default".
The claim has been made jointly against RBS Securities, RBS Financial Products Inc and Financial Asset Securities Corp.
Legal documents filed by Assured and seen by The Herald stated: "As has become apparent as Soundview losses have mounted and the unlawful securitisation practices of RBS and other financial institutions have been exposed, defendants were jointly engaged in an unmitigated effort to acquire and securitise as many mortgage loans as possible, intentionally passing undisclosed investment and credit risks to investors and other market participants such as Assured.
"Both RBS Financial Products and RBS Securities were well aware, and Assured was not, that WMC was among the most shoddy and irresponsible mortgage loan originators the RMBS (residential mortgage-backed security) industry had ever seen."
Assured said that review of 1482 of 1593 mortgage loans originally included, 1376 - 93% - violated one or more of RBS's representations to the company.
RBS refused to comment on the case. However, representations made in a motion by RBS indicates the blame lies with WMC, the now closed mortgage arm of US-based General Electric, which had been the subject of an FBI and Justice Department investigation.
Documents stated: "The crux of the complaint is that WMC and the loan borrowers themselves, not RBS, made various alleged misrepresentations in connection with the origination of the loans.
"There are no allegations that RBS did anything other than restate information provided by WMC or the borrowers, and thus there are no allegations establishing RBS' intent to deceive Assured."