Many holidaymakers to Britain decided to stay London and soak up the atmosphere surrounding the world's biggest sporting spectacle causing Scotland to lose out.
Latest official figures show travellers spent 114,000 fewer nights in the Scotland between July and September last year compared to the same period in 2011 – a drop of 12%.
Overseas visitors also spent £53 million less in the summer of 2012 than the previous year.
Both figures were considerably worse than other parts of the UK, suggesting Scotland was particularly hard hit by the Games, as well as dire weather and a sluggish economy.
Scotland's national tourism agency said it was disappointed, but said the wider picture was positive.
Mike Cantlay, chairman of VisitScotland, said: "Times have been undeniably tough for businesses over the summer, particularly in certain sectors such as outdoor attractions and activities.
"We all know the summer was a total washout except in the north-west, with many event cancellations, and that the Olympics impacted on domestic visits outside of London.
"This is such a shame because the first part of the year was a huge success despite the economic gloom – evidenced by a 3% increase in spend in the past 12 months, and a tourism industry that was geared up for making the summer a success."
The Office of National Statistics released a tranche of tourism data yesterday, which included some good news for Scotland.
Spending from oversees travellers for the year to September 2012 is up 12% when compared to the previous year, while the number of foreign visitors also increased.
But the healthy longer-term outlook failed to hide a dismal summer for Scottish tourism.
London experienced a £370m boost from foreign spending last summer compared to 2011 as tourists bought Olympics and Paralympics tickets.
Scotland, on the other hand, saw a £53m decrease on 2011. England was up £490m and Wales up by £1m.
Similarly, it was Scotland that was worst hit by the drop in visitors in 2012. Last summer, tourists spent 12% fewer nights north of the Border than in 2011, compared with a 5% drop in London, a 3% drop in England and a 9% rise in Wales.
Professor John Lennon, director of the Moffat Centre for Travel and Tourism Business Development, said: "London is a gateway to the UK. An awful lot of Scottish tourism will come in through London particularly the international stuff."
He suggested the spin-offs from films located in Scotland such as the animation Brave and James Bond blockbuster Skyfall could see visitors return.
Willie MacLeod, executive director of the Scotland branch of the British Hospitality Association, agreed the Olympics "kept people away from Scotland" but predicted a healthy surge in tourism at the end of 2012.
Tourism Minister Fergus Ewing said: "There is no denying conditions were tough this summer."
However, the Royal Yacht Britannia berthed at the Ocean Terminal in Leith, Edinburgh, bucked the trend with its strongest performance in a decade, with 123,111 visitors – a rise of 5%.
And Scotland's two national parks are to receive £2.9m for projects marking the start of a year-long initiative celebrating the great outdoors.
Cairngorms National Park, which attracts 1.4 million visits each year and contributes £115m to the local economy, has received an additional £2.31 million. Loch Lomond and Trossachs, which has four million visitors giving back £154m to local commerce, has received £580,000. Both sums are earmarked for projects that will bring significant economic, environmental and tourism benefits.
It coincides with the The Year of Natural Scotland scheme, led by VisitScotland.