On a good day, the journey from Glasgow to Mount Stuart, the ancestral home of the Earls and Marquesses of Bute, can be one of the loveliest in Scotland.

A train from Central and the ferry from Wemyss Bay, and a scenic journey across water later, you are in Rothesay. A short taxi or bus ride, and you are at the 19th-century palace of Mount Stuart, the beautiful Gothic revival home, built by John Patrick Crichton-Stuart, 3rd Marquess of Bute.

The ancestral home, one of the most unusual as well as most spectacular in Scotland, is a treasure-trove of art from 300 years of history: packed with art, artefacts, tapestries, and remarkable and unusual architecture and design.

More than 165,000 people have visited since it opened to the public in the 1990s, and the exquisite central Marble Hall is worth the trip alone. But should you want to see some of the cream of those art collections, considered one of the finest private collections of old Masters in the UK, you do not have to make that trip any more – they are moving to Edinburgh.

From next week, 19 of the finest paintings in the Mount Stuart collections will be loaned for at least three years to the National Galleries of Scotland. Initially there will be an exhibition, simply titled Masterpieces from Mount Stuart: The Bute Collection, which will run from 18 May to 2 December, free of charge, at the Scottish National Gallery. The paintings will then enter the collections on The Mound for at least another three years, perhaps more.

They are very fine pictures too, the bulk of them bought by the 3rd Earl of Bute, who also happened to be the first Scottish Prime Minister of the UK in 1762-3. The current Marquess of Bute is the 7th, John Crichton-Stuart, known as Johnny Dumfries in his time as a racing driver, and it was his decision, and well as his families, to enter into the pact with the National Galleries which will see them moved to the capital en masse for the first time since a show in 1949.

The excellent foreword to the collection catalogue is written by the Marquess's brother, Anthony Crichton-Stuart, a noted art historian. He says that the collaboration between Mount Stuart, which opened to the public in 1995, and the National Galleries has long been mooted, perhaps since the establishment of the Mount Stuart Trust in 1989. "I think it is fair to say this is something that has always been on the cards. There has always been dialogue, it has always been thought about and now really does seem like the right time," he says. The 3rd Earl, he says, collected on a "very grand scale".

His name was John Stuart and lived from 1713 to 1792, and he collected heavily, along with his son, also John, the 4th Earl and the 1st Marquess, who lived from 1744 to 1814. "It was at that time that the basis of the collections were really formed," Mr Crichton-Stuart says. "Other members of the family collected but that is when the serious work was done. The paintings were bought to his particular taste. He had been educated at the University of Leiden and that may well have been a catalyst for the quality of the Dutch pictures."

The works include leading Dutch painters of the 17th century, including Jan Steen, Pieter de Hooch and Jacob Jordaens. There are two rural landscapes by Aelbert Cuyp (1620-1691), an artist popular with British collectors in the 18th century, a winter landscape by Aert van der Neer (1604-1677), and Jacob van Ruisdael's Mountain Landscape with a Waterfall (c.1665-70), which will be shown alongside a winter view of Amsterdam by the artist, which is appearing for the first time in a public exhibition.

Among the portraits on show will be Jacob Jordaens's picture of a girl, from the late 1630s, and Joos van Cleve's Portrait of a Lady (c.1530).

Hildegarde Berwick, Mount Stuart's head of collections, says: "You can see so much art on the walls that haven't gone to the National Galleries – it is an indication of the quality of the collections. The paintings that have been loaned will be integrated with the collection at the National Galleries for at least three years and after that, we shall see. The quality of the works is so high here that there are some other areas that we can explore."

The relationship has another link – Mount Stuart's architect was Sir Robert Rowand Anderson, who later designed the Scottish National Portrait Gallery in Edinburgh. You can clearly see the same architectural mind at work, the central hall of both buildings being similar, in particular.

Mr Crichton-Stuart said the collaboration is therefore "particularly appropriate" and adds: "The 3rd Earl of Bute himself had strong ties to Edinburgh, and he was born there in a house about five minutes' walk from the current gallery site. There is of course a further family connection to Edinburgh with Bute House in Charlotte Square being the official residence of the First Minister of Scotland."

The organisation of the loan was a team effort, with key contributions from Connie Lovel, the chief executive of the Mount Stuart Trust, as well as Ms Berwick who worked with the National Galleries of Scotland team led by Dr Tico Seifert, a senior curator at the NGS. The pictures themselves were collected from about 1749 to 1792 by the 3rd Earl, with additions from his son and the later 4th Marquess (1881–1947) who, in 1934, bought Pieter Saenredam's The Interior of St Bavo's Church in Haarlem, which is now permanently in the Scottish National Gallery, Edinburgh, as well as Jacob van Ruisdael's Winter View of the Hekelveld in Amsterdam.

But it is the 3rd Earl who founded the core of this collection. Prime Minister for King George III for only 317 days, he became deeply unpopular, notably for a tax on cider which he proposed. However he ended the Seven Years' War, negotiating the Treaty of Paris with France in 1763, and added huge colonial territory to the British Empire. He bought a lot of the collection with a friend and dealer, Captain William Baillie (1723–1810), an Irish captain of the Dragoons. His knowledge was of Dutch painting, and he made extensive purchases for Bute of these works, as well as Flemish artists such as Rubens, Van Dyck, Brueghel and Teniers.

There are hundreds of works of art at Mount Stuart, and it is easy to imagine future loans in the future. Indeed, this is not the first time a collaboration has been suggested or discussed. The 2nd Marquess of Bute (1793-1848) had discussions to loan the pictures which had hung at the family's former residence at Luton Hoo in Bedfordshire. This was in 1845, two years after a disastrous fire at Luton Hoo and at that time, the Royal Scottish Academy in Edinburgh was considering building a new gallery to house the Bute collection.

A few years later, the Royal Institution and the Board of Manufacturers – the predecessor of the National Gallery of Scotland – made a similar proposal. In 1847, they applied to exhibit pictures from the Bute estate in the Royal Institution Galleries in Edinburgh: but the offer was refused and the Bute Collection was moved to the family homes in Scotland. Now they are, at last, making a lengthy stay in the capital. Anthony Crichton-Stuart adds: "[Those plans] didn't happen, but it's fitting that the pictures have now ended up in Edinburgh 170 years on."

There is another legacy of the relationship, which is again bearing fruit. He adds: "The 1949 show in Edinburgh, the largest display of works from the Bute collection until now, is also significant as the fledgling Edinburgh Festival at the time was very underfunded. So our grandfather's agreement to loan the pictures gave the Edinburgh Festival a blockbuster exhibition at relatively little cost, thereby helping ensure the early success and viability of an arts festival that has subsequently grown into the largest and best in the world."

Masterpieces From Mount Stuart: The Bute Collection is at the National Gallery, Edinburgh from May 18 to December 2.