New figures from the Association of Leading Visitor Attractions (ALVA) also showed that Edinburgh Castle was the most popular paid attraction outside London, with 1.42 million visitors, following an increase of 15% in attendances.
Glencoe, cared for by the National Trust for Scotland, also saw a large increase in tourism, with a significant 41.7% rise in visitors, perhaps due to being heavily featured in the recent James Bond movie, Skyfall.
Kelvingrove Art Gallery and Museum in Glasgow saw a small rise in visitors of 0.2% up to just over one million visitors, making it the third most popular site in Scotland and the 24th in the UK, but the Riverside Museum in Glasgow experienced a drop of more than 26% to just over 740,000.
Overall, ALVA saw its members' visitor figures rise by an average of 6% on 2012, with Scotland posting an average 5% improvement, compared to 12% in London.
The British Museum remained the most popular visitor attraction for the 7th year running with a 20% increase in its visitor numbers to 6,701,036 - making it the museum's most successful year on record.
The National Museum of Scotland saw a drop of 7% in its visitors despite leading the way north of the border, its figures taking into account both free and charged admissions.
Dr Gordon Rintoul, director of National Museums Scotland, said: "We are pleased and proud of these results.
"Being confirmed as the most visited attraction outside London for the third year running demonstrates that we continue to capture the imagination of visitors with compelling exhibitions, imaginative public programmes and internationally important collections.
"We have welcomed over five million visitors to the National Museum of Scotland since its reopening in summer 2011."
Edinburgh Castle was the number one paid-for visitor attraction in Scotland last year and the most visited paid-for UK attraction outside London.
An increase in visitor numbers was also recorded at Historic Scotland's Stirling Castle (up 2% to 411,726) and Urquhart Castle (up 5% to 310,446).
Fiona Hyslop, the culture secretary, said: "Edinburgh Castle continues to captivate visitors from around the world and I am delighted that it has welcomed more than 1.4 million visitors for the first time, which is a significant milestone.
"Scotland's heritage sector continues to play an important role, supporting 60,000 jobs, contributing over £2bn to our economy and continuing to enthral visitors from home and abroad."
ALVA's 56 members are among the UK's most popular and important museums, galleries, palaces, castles, cathedrals, zoos, historic houses, heritage sites, and gardens.
A spokesman for Glasgow Life said that overall its nine civic museums performed "exceptionally well" with more than 3.1 million total attendances in 2013.
The Burrell Collection, which is due to close for a major revamp, saw a rise of 4.7% to more than 193,000 visitors.
He said: "Almost every museum has seen an increase in visitors, thanks to a broad range of exhibitions, such as the record breaking Jack Vettriano Retrospective at Kelvingrove and Masters of Impressionism at the Burrell Collection. "It was always expected that the Riverside Museum would record a drop in visitor numbers after enjoying massive success in its first few years."
A spokeswoman for the National Galleries of Scotland said its visitors numbers are down 8.9% for 2013 against 2012 figures, but up by nearly 15% on 2011.