Thousands of Scottish firms may demand compensation from their banks after the City regulator found them guilty of mis-selling complex products to small business customers.

The Financial Services Authority (FSA) has identified mis-selling of interest rate hedging products in 90% of the 173 cases studied in a pilot review, and said a "significant proportion" of those businesses should receive redress.

It ordered the four biggest banks – RBS, Lloyds, Barclays and HSBC – to review all their sales of interest rate swap agreements (IRSAs) and other hedging products on a case-by-case basis over the next six to 12 months.

Confirming the scale of the scandal, highlighted by The Herald, the FSA said most small businesses had been unlikely to understand the risks of what were in many cases "absurdly complex products".

However, although up to 2500 Scottish small businesses were sold hedging products, only a small proportion had the most complex versions which were studied in the pilot cases, and the FSA warned its findings "may not be representative of all sales".

The banks' eventual liability is likely to be a fraction of the £12 billion in the PPI scandal, but may be at least double their current £700m provisions for IRSA compensation.

RBS said: "We will work with our customers to ensure a fair and timely resolution of these issues."

IRSAs were promoted heavily by banks before the financial crash as a protection for businesses against rising interest rates. But small businesses were not aware that this was a new breed of loan, where if interest rates fell sharply they would not only miss out on lower costs and perhaps see their rate go up, but would also have to pay a huge penalty to exit the contract.

That was because behind the sale the bank created a complex derivative or "swap" agreement, which it would immediately sell on in the money markets, banking a healthy profit. When interest rates crashed, the banks lifted rates on many agreements and set "break fees" which could amount to 20% or more of the loan, to ensure they did not lose any of their profit.

The Bully Banks campaign group says 89% of its 1000 members were never given an illustration of the break fee in the event of a substantial fall in interest rates, while 73% say their bank made the IRSA a condition of continued borrowing. Both scenarios would be a breach of FSA rules.

The first UK test case in the Court of Session last year saw small property firm Grant Estates lose its case against RBS largely because the bank's contract said it had not given any advice. But the FSA says such contracts did not protect banks from conduct-of-business rules that required them to understand their customers' level of expertise and spell out breakage costs.

The regulator has also widened the definition of an "unsophisticated" customer to include in the review small businesses with seasonal spikes in employment.

Paul Adcock, a director of Bully Banks, said the mood was one of cautious optimism. He added: "It addresses a lot of the issues we have been concerned about, though there are still some unresolved."

Martin Wheatley, chief executive designate of the new Financial Conduct Authority which will shortly replace the FSA, said: "Where redress is due, businesses will be put back into the position they should have been without the mis-sale. But it is important to remember that this review is firmly focused on the particular circumstances of each sale."

Andy Willox, Scottish policy convener for the Federation of Small Businesses, said: "Businesses caught up in this scandal will be pleased the FSA has recognised the vast majority of them were sold products that simply weren't right for them."

He said there was a worrying lack of clarity on the issues of redress, consequential losses, and the appeals process, as well as no requirement on banks to suspend a firm's IRSA payments during the review.

Simon Jaquiss, an adviser to the QA Legal group of Scottish and English lawyers, said: "This appears to kick the ball further down the road, potentially allowing banks to escape what might in legal terms be fair and proper redress by allowing them six to 12 months to come up with their assessment. It also does little to allay the fears of those with simple products who form the vast majority."

Cat McLean, partner with Edinburgh law firm MBM Commercial, said firms would need expert help to engage with the review, and should keep open the option of legal action, especially in view of the five-year time bar on bringing an action.