The Financial Conduct Authority (FCA) said Clydesdale had failed to clearly spell out customers' rights following the bank's miscalculation on the repayments on more than 42,500 mortgages.
In total, the earlier blunder meant a £21.2 million shortfall in Clydesdale mortgages, with customers who underpaid left with outstanding mortgage balances higher than they should have been. The shortfalls range from less than £20 to more than £18,000, with an average of £970.
Clydesdale, which is owned by National Australia Bank, sent letters to customers in 2009 which suggested that they had no alternative but to bring their repayments up to date.
But many customers could have rejected demands to repay the shortfalls caused by Clydesdale's calculation errors.
Clydesdale has agreed to compensate those who were adversely affected, resulting in a total cost to the bank of about £42 million.
Customers who were left with shortfalls will automatically be compensated and the bank is writing to all customers who were affected by the blunder. Mortgage-holders do not have to do anything until they are contacted by Clydesdale to explain what the next steps are.
David Thorburn, chief executive of Clydesdale, said he was "very sorry" that the situation had not been handled properly.
He said: "We should have made it clear at the time that this was entirely our fault and that some customers may be entitled to compensation.
"Our priority is to fix this for customers as quickly as possible and they will each receive a letter explaining how we will make this right for them."
Some 14,000 customers will see an "immediate resolution", including a full apology and a refund within the next 48 hours, he said.
The vast majority of other customers with more complicated cases should hear from the bank in the next couple of weeks.
Some customers may also be entitled to compensation for extra charges they have had to incur as a result of the original mistake, such as bank charges.
Consumers can still take their case to the financial ombudsman, which resolves disputes between consumers and financial institutions, if they are unhappy.
The bank's "lack of clarity" was made worse by poor instructions given to call handlers for dealing with customers who called to complain, the FCA said.
The regulator said the fine would have been higher had it not been for Clydesdale's redress scheme, and the bank received a 30% discount for settling at an early stage.
Tracey McDermott, the FCA's director of enforcement and financial crime, said: "For most people, mortgage payments are their biggest monthly outgoing and we all budget on the assumption that the information our mortgage lender gives us about what we need to pay is correct.
"Here Clydesdale failed in that basic duty and, when it discovered the problem, sought to pass all of the consequences on to its customers - expecting them to remedy mistakes which were entirely of Clydesdale's making."
Ms McDermott said firms must put the interests of consumers at the heart of their business to restore trust and confidence in financial services.
She continued: "Clydesdale is today paying the price for its decision to put its bottom line ahead of the need to ensure its customers were treated fairly."
Consumer information about the bank's compensation arrangements has been placed at cbonline.co.uk.