The customers were not paid redress within 28 days of receiving a decision letter and almost 9,000 had to wait more than six months for their compensation, the Financial Services Authority (FSA) said.
The failings relate to Lloyds TSB Bank, Lloyds TSB Scotland and Bank of Scotland, leading to a total fine of £4.3 million.
The regulator said it had imposed a large fine because Lloyds' redress payments systems fell "way below the standard the FSA expects".
Nearly one quarter of 582,206 decision letters agreeing to pay PPI compensation between May 2011 and March 2012 did not result in payouts being made within the target of 28 days. Under FSA rules, such compensation should be paid out promptly.
The regulator said that around 87,000 customers had to wait more than 45 days, 56,000 over 60 days, 29,000 past 90 days and 8,800 in excess of six months.
Some 24,589 payments were mistakenly dropped out of the process and later came to light as a result of customers calling to chase the vanished payments and through media attention, the FSA said.
When customers tried to chase up redress payments they had not received, deficiencies in its process meant the group was unable to fast-track the payments, tell them when they would get their money or explain the delays, the FSA said.
The FSA said that Lloyds had not planned sufficiently "at the outset" and its systems were not able to process the large volumes of PPI payouts in a timely way.
Staff did not have the knowledge and experience to make sure that the system worked properly and the tracking of payments was ineffective, the watchdog said.
Lloyds apologised to customers and acknowledged that it had underestimated the scale of the PPI scandal.
The banking giant said "almost all" customers who were due redress during the review period have been paid in full and customers have not been left out of pocket because of the delays.
Lloyds said in a statement: "When we took the lead in 2011 to compensate customers on PPI, we had not fully anticipated the volume of complaints to be processed at the outset and experienced some administrative errors as we scaled up our systems and processes.
"We acknowledge that this led to some customers not being compensated on time and we apologise to those customers whose payments were delayed.
"It is important to note that almost all customers who were due redress during the review period have now been paid in full."
Martin Lewis, founder of consumer help website MoneySavingExpert, pointed out that today's fine is just a fraction of Lloyds' budget for compensating customers who have been mis-sold PPI.
He said: "In truth this fine is paltry. £4.3 million is just 0.08% of its total PPI redress budget of £5.3 billion, a relatively minor cost of scale - and a fraction of the money it was late to pay. So it's questionable how big an impact it'll have."
The group agreed to settle with the FSA at an early stage and qualified for a 30% discount, otherwise it would have received a £6.16 million fine.
It is understood that less than 1% of the Lloyds customers who faced delays are still waiting for payment while the banking giant pinpoints exactly where the payments need to be made.
Some of these customers may have had a change in circumstance which makes it more difficult to work out exactly where the payment needs to go, but as with PPI compensation payouts generally they will be paid interest, meaning they are not left out of pocket.
Lloyds has undertaken a review of PPI redress payments and has improved its processes to address the failings identified, the FSA said.
Last month, the FSA fined the Co-operative Bank £113,300 for delaying valid PPI compensation claims "for no good reason", affecting a significant proportion of 1,629 complaints.
The Co-op has said it is confident that there will be no repeat of the delays and no one lost out financially as a result.
The Financial Ombudsman Service (FOS), which recently said that it handled a record 11,000 complaints a week about PPI mis-selling in the last three months of 2012, has criticised banks for dragging their heels over the issue and subjecting customers to inconvenience.
The FSA's figures show that a total of £8.4 billion has been made in PPI payouts by the industry over the last two years since January 2011.
Around £14 billion has been put aside by banks to deal with PPI mis-selling complaints, but some analysts have suggested this could soar by around another £10 billion as the scandal has surged beyond expectations.
Tracey McDermott, the FSA's director of enforcement and financial crime, said: "The industry let customers down badly in relation to the sale of PPI.
"The significant volume of complaints is a product of LBG (Lloyds Banking Group)'s own failings and the least customers can now expect is that redress, when it is due, will be paid promptly.
"In short, LBG's PPI redress payment systems fell well below the standard the FSA expects, and the size of this fine reflects how seriously we view these breaches."