The watchdog said that, between October 2009 and January 2012, ScottishPower provided customers with inaccurate estimates of annual charges and comparisons with their current supplier if they switched to the firm and failed to adequately monitor its sales staff.
The company has set up a £1m compensation fund for those affected, while the remaining £7.5m will be paid to more than 140,000 vulnerable customers, who will automatically receive payments of about £50 each by December.
ScottishPower will write to 336,000 households that may have been mis-sold. Customers who believe they were affected can also contact the group on 0845 0303048 or online at www.scottishpower.co.uk/salescompensation.
Trisha McAuley, Scotland director of Consumer Futures, a UK Government body set up to represent consumers, said: "Mis-selling was the original sin of energy competition and it is right compensation is made to customers affected by it. But it is unfortunate it has taken so long to resolve.
"Ofgem and energy suppliers need to move much faster in bringing these issues to a close - long, drawn out investigations do not help anybody."
Ofgem said that, despite the sales practice failures, it found no evidence ScottishPower deliberately set out to mis-sell to customers.
ScottishPower apologised "unreservedly" to those affected and said it had taken steps to address its failures. The group stopped doorstep selling in June 2011 and has overhauled training and monitoring procedures for all telesales staff.
Neil Clitheroe, the firm's chief executive of energy retail and generation, said: "We accept Ofgem's findings and apologise unreservedly to those customers affected.
"This arose as a result of new regulations in 2009.I am sorry we did not implement these properly at that time."
The investigation into ScottishPower comes as part of a wider market inquiry into mis-selling across the energy industry, which has already seen three investigations concluded.
Ofgem said it decided not to levy a fine against ScottishPower as the £8.5m customer payment package agreed by the firm would be "of greater benefit to energy customers than if a substantial penalty was imposed".
Sarah Harrison, Ofgem's senior partner in charge of enforcement, said: "Ofgem welcomes ScottishPower's recognition of its failure to comply with the energy sales rules which are there to protect the consumer. This is an important step forward and demonstrates a commitment by ScottishPower towards re-establishing consumer trust."
Shadow energy secretary Caroline Flint said the fine was "more evidence that Britain's energy market is broken".
But Energy Secretary Edward Davey said: "This is a clear, strong signal that energy companies should not expect to get away with bad practice. We are giving Ofgem powers that force energy companies to make direct payments to consumers hurt by these kinds of activities, and backing up Ofgem's reforms so consumers get a simpler, fairer deal."