The Burrell Collection in Pollok Country Park, Glasgow was given the distinction by Historic Scotland because it is an example of 1970s design.
The £16.2 million building houses around 8,000 paintings and objects collected by shipping magnate Sir William Burrell that was left to Glasgow after his death.
The museum opened to the public in 1983 and attracts around 200,000 visitors every year.
Culture Secretary Fiona Hyslop said: "The Burrell Collection is one of Glasgow and Scotland's most impressive buildings of its period. This is a fantastic building that not only houses the internationally renowned collection of art and antiquities from across the world but is itself a masterpiece of structural design."
Only 1% of Scotland's buildings are listed.
Councillor Archie Graham, chair of Glasgow Life, the council firm that runs the city's museums, said: "It says everything about the Burrell Collection building that is regarded so highly in its own right and is often named as one of Scotland's best.
"The A-listing from Historic Scotland allows us to celebrate this world-famous building as proposals are brought forward for a refurbishment which will secure it for future generations."
Repairs are needed to the roof of the building and Glasgow Life says it wants the art collection to tour the world in order to fund the work.
Sir William left the collection to the city on the condition that it was not loaned overseas in case it is damaged in transit. But the museum board insists that improvements in transport may allow it to relax Sir William's restrictions.