The watchdog said that, between October 2009 and January 2012, ScottishPower provided customers with inaccurate estimations of annual charges and comparisons with their current supplier if they switched to the firm and failed to adequately monitor its sales staff.
ScottishPower has set up a £1 million compensation fund for those affected, while the remaining £7.5 million will be paid to more than 140,000 vulnerable customers, who will automatically receive payments of around £50 each by December.
ScottishPower will write to 336,000 households that may have been mis-sold.
Customers who believe they were affected can also contact the group on 0845 030 3048 or online at www.scottishpower.co.uk/salescompensation.
Ofgem said that, despite the sales practice failures, it found no evidence that ScottishPower deliberately set out to mis-sell to customers.
ScottishPower apologised "unreservedly" to those affected and said it had taken steps to address its failures.
The group stopped doorstep selling in June 2011 and has overhauled training and monitoring procedures for all telesales staff.
Neil Clitheroe, ScottishPower's chief executive of energy retail and generation, said: "We accept Ofgem's findings and we apologise unreservedly to those customers affected.
"This arose as a result of new regulations which were introduced in 2009. I am sorry to say that we didn't implement these properly at that time."
The probe into ScottishPower comes as part of a wider market inquiry into mis-selling across the energy industry, which has already seen three investigations concluded.
SSE was fined £10.5 million in April for "prolonged and extensive'' sales practice failures.
Ofgem is still conducting two ongoing mis-selling investigations into npower, which was launched in 2010, and into E.ON, which was launched last year.
ScottishPower said that, of the households affected by mis-selling, around 50,000 potentially lost out financially and could be due compensation.
It estimates compensation payments will vary from between £5 and £30 for each gas and electricity service provided to households.
Ofgem said it decided not to levy a fine against ScottishPower as the £8.5 million customer payment package agreed by the firm would be "of greater benefit to energy customers than if a substantial penalty was imposed".
It added that breaches of its rules on energy firm sales activities were vital to protect consumers.
Ofgem warned: "Consumers can suffer financial detriment if they do not benefit from savings that they are led to expect - they may switch to a more expensive deal and competition suffers as consumers lose faith with the market and the value of switching energy supplier."
Sarah Harrison, Ofgem's senior partner in charge of enforcement, said: "Ofgem welcomes ScottishPower's recognition of its failure to comply with the energy sales rules which are there to protect the consumer.
"This is an important step forward and demonstrates a commitment by ScottishPower towards re-establishing consumer trust."
Ofgem fined EDF Energy £4.5 million in March last year for misleading sales claims as part of the industry probe.