Now Abbotsford House, where Scott penned some of his most famous works such as the Waverley Novels, Rob Roy and Ivanhoe, is facing a new challenge over plans to site houses nearby.
The trustees of the house are on collision course with Scottish Borders Council over a proposal to designate a greenfield site for housing just yards from Scott's former home.
The baronial mansion, which re-opened last year after the upgrade, is now established as arguably the leading tourist attraction in the Borders.
However, the council has included Netherbarns on the outskirts of Galashiels, Selkirkshire, as a potential site for 45 homes in its local development plan, which will influence planning decisions until 2024.
The council is currently undertaking a public consultation on the programme and will consider all responses at a meeting on June 24.
An objection has already been lodged by the chairman of Abbotsford House Trust, Lord Sanderson of Bowden.
Two previous public inquiries have deemed the 7.5-hectare greenfield site - directly across the Tweed from Abbotsford - unsuitable for housing.
In a letter to the council, Lord Sanderson said his board of trustees were unanimous in objecting to the Netherbarns allocation and were disappointed it was being considered again.
He said: "It is our case that any development at Netherbarns will have wholly inappropriate impact, visually and in many other ways, on Abbotsford.
"It is well known that Abbotsford has undergone an £11.5 million capital works programme over the last two years including complete refurbishment of the house, the building of a new visitor centre, shop and restaurant and new interpretation facilities inside and outside the house.
"Every aspect of Abbotsford as a visitor attraction has now been accredited with five-star status."
Lord Sanderson, a former Scottish Conservative Party chairman, revealed there were 70,000 visitors in 2013, with 33,000 of these paying to go round the house, despite it only being open half-way through the year after the Queen performed the opening ceremony in July.
He says one-third of these visitors came from overseas and 73 per cent from outwith the Borders.
Lord Sanderson continued: "Abbotsford is the jewel in the crown of the tourism assets of the Borders and with the advent of the Borders railway terminal at Tweedbank in 2015, the potential for further growth is considerable.
"It thus seems to the trustees perverse that the council planning department should now be considering any development which could possibly impact on that investment for the future."
The Netherbarns site was purchased by Kelso builders M & J Ballantyne from the Scott family for a reputed £2m in 2005.
The following year the company was granted consent by the council's planning committee to build 79 detached houses there.
But as the objectors included Historic Scotland, which claimed development would have an adverse impact on the views from Abbotsford, a public inquiry was held, determining in 2007, that the Ballantyne application should be refused. Another inquiry ruled the site should not be designated for housing in the council's current local plan, which is about to be replaced.
If the new local development plan includes 45 houses for Netherbarns with potential for "educational facilities" also on the site, it would be down to a reporter from the Scottish Government to arbitrate later in the year.