More than half of that cash has been paid out by Scotland's largest local authority, Glasgow City Council.
Compensation payments to drivers by the council have totalled £1,821,576 over the period 2007-08 to 2011-12, according to figures which were obtained by the Conservatives using freedom of information.
Across Scotland, drivers have received £3,558,201 from local councils to compensate for damage caused by potholes over the five years.
The amount is increasing, with £1,232,908 paid out last year, compared to just £342,476 in 2007-08.
It comes after a survey of members of the AA motoring organisation found those in Scotland were most likely to report pothole damage to their cars, with 44% saying their vehicles had suffered damage in the last two years.
The Tories are now urging the SNP administration to prioritise work to repair road surfaces.
Conservative transport spokesman Alex Johnstone said: "Potholes are a real problem across Scotland and you can see this from these figures.
"If the SNP dropped its anti-car agenda and actually tried to improve the roads, it could help save the councils some money in future."
While Glasgow City Council paid out more than £1.8 million over the five-year period, Orkney Islands Council compensation payments totalled just £244.
But Mr Johnstone said many drivers could have had their cars damaged by potholes but not claimed cash back from the relevant council.
He said: "No-one wants to sit down and take action against the council because of damage the roads have done to their car, because it's a lengthy and complex process.
"But times are tight and a trip to the garage is rarely a cheap one.
"These figures won't even include the many people who simply can't face the red tape and pay for the damage themselves."
He added: "It's expensive enough being a motorist in Scotland today without the shoddy state of the country's roads adding to the problem."
A Scottish Government spokeswoman said: "The maintenance of local roads is a matter for local authorities, who received almost £11 billion in Scottish Government funding this year.
"Additionally, through our £250 million annual investment, Scottish ministers continue to deliver an extensive maintenance and improvement programme on our major A-class roads and motorways, with road safety a key priority - despite a 26% cut to Scotland's capital budget as a result of Westminster government cuts."
Councillor David O'Neill, president of local government body Cosla, accused the Tories of "opportunistic council bashing" as he stressed authorities only pay compensation when instructed to do so by their lawyers.
Mr O'Neill said the Conservatives "should perhaps remember that councils are made up of all political parties and as they returned an increased number of councillors in the May elections, they are having a pop at themselves to a large extent".
He added: "The bottom line is that in the modern world we now occupy, there is far more of a compensation culture and people are often actively encouraged to pursue claims through no-win no-fee lawyers.
"However, that said, councils only pay compensation when they have been instructed to legally.
"They do not pay it willy-nilly and to suggest otherwise is both wrong and misleading."
Contextual targeting label: