Mount Stuart House on Bute, seat of the marquesses of Bute, is a popular tourist destination and rightly so.

Commissioned in the late 1870s by the 3rd marquess, then Britain's pre-eminent architectural patron, it was built by Sir Robert Rowand Anderson in imposing red sandstone and replaced the Georgian pile which burned down in December 1877. It was the first home in Scotland to have electric lighting and a heated swimming pool, and its innovations continue in the 21st century with an award-winning visitor centre and a rolling series of exhibitions in the grounds by A-list artists such as Turner Prize nominee Nathan Coley.

Within Mount Stuart's walls, however, it's A-list artists of previous centuries who dominate, proof that the 3rd marquess wasn't the only member of this ennobled Scottish family to fancy himself as a patron of the arts. Both his grandfather, the 1st marquess, and in particular his great-grandfather, the 3rd Earl of Bute, were keen art buyers and important taste- makers, and together they assembled Britain's greatest collection of 17th-century Dutch painting.

A selection of this work has now been loaned to the Scottish National Gallery where it hangs in a small room off the cavernous main thoroughfare, as dense a collection of Old Masters as it's probably possible to achieve. In all, 19 paintings have headed east in the largest public display of works from the estate since a 1949 Edinburgh Festival exhibition. The only other view of them in Scotland prior to this was an 1884 exhibition in Glasgow.

The centrepiece is Willem van Haecht's massive allegorical Art Cabinet With Anthony Van Dyck's Mystic Marriage Of St Catherine, painted around 1630 and wrongly ascribed to Frans Francken the Elder when it showed in Glasgow. Appropriately, its depiction of a room lined entirely with paintings – some real, some imaginary – replicates the experience of this exhibition, though hopefully the National Gallery won't have to deal with anything as sinister as the scene van Haecht depicts in the background of his work: two figures with donkey heads smashing statues and destroying paintings. A devout Catholic from Antwerp who helped restore altarpieces destroyed there during the Reformation, it's thought van Haecht intended this scene as an allegory about Protestant barbarism. It certainly adds another air of mystery to this beguiling work.

Facing the van Haecht and flanking the entrance are two of the smallest paintings, enigmatic domestic scenes by Cornelius Bega. Small they may be but their subject matter – a drunk man in a tavern, and a young woman and an older man singing in a dark, untidy room – are emblematic of the everyday themes to which these Dutch artists of the mid-17th century were drawn.

To the patrician eye of art historian Kenneth Clark, their work is "the first visual evidence of bourgeois democracy", and there's nothing here to make you think he was wrong. With the exception of Bartholomeus Breenbergh's 1654 painting Joseph Distributing Corn In Egypt, there isn't a whiff of the biblical or the religious, and the closest we get to a church is a sight of Haarlem's late-gothic St Bavo, poking up from behind the portico of the town hall in Gerrit Berckheyde's study of Haarlem's market place. In fact there are as many windmills on view as there are places of worship.

Everywhere you look, then, the feeling is of the artist as documentarian. Look at the works by Pieter de Hooch, Jan Steen, David Teniers the Younger and Gabriel Metsu, and you see men and women arguing over disputed bills, gambling at cards, flirting, posing or simply feeding a dog. These are real people, real places and real lives.

The least impressive works, if only in the sense that they jar with these themes of civic life, are a bucolic pair by Aelbert Cuyp. One shows a horse, a cowherd and some cattle; the other a group of five cows watering by an estuary. They were acquired by the 3rd Earl of Bute before 1776 but, if nothing else, they illustrate the peer's importance as a tastemaker. Cuyp was little known in Britain before the earl began acquiring his work, but he soon became a favourite among collectors as a result of the peer's influence.

Masterpieces From Mount Stuart carries the unavoidable tang of privilege and wealth, but also evident is something even money can't buy – a large degree of inherited good taste. It's worth noting as a postscript, however, that even this breathtaking collection is not what it once was. The family sold as well as bought, and in one auction in the spring of 1794, the 3rd Earl disposed of a collection of prints, drawings and etchings which included 138 works by Durer and 272 by Rembrandt. That sale, for the record, raised £2777, 10 shillings and sixpence.

Masterpieces From Mount Stuart: The Bute Collection is at the Scottish National Gallery until December 2