It was snubbed by the Scottish Government during this summer's Olympic Games.

But the cream of British acting talent is flocking to the Scotland Office in London, as the base for Scotland's civil servants in England develops a new life as a film set.

Dover House has long been known as one of the most desirable addresses in Whitehall.

The Georgian mansion was notorious as the scene of Lord Byron's scandalous affair with Lady Caroline Lamb.

Both Tony Blair and Margaret Thatcher are rumoured to have considered decamping from Downing Street, during their terms as prime minister, to take advantage of its sweeping views.

In recent months the old building has been hosting film stars and camera crews.

A number of scenes for the James Bond thriller, Skyfall, were shot there and elsewhere in Whitehall.

The building was also chosen to house Daniel Craig's green room – for the 007 actor to use between takes.

The atrium of the building doubled as part of the Foreign Office in the recent Channel 4 political thriller Secret State, starring Gabriel Byrne as an under-pressure prime minister.

The office of Lord Wallace of Tankerness, the former Deputy First Minister of Scotland and now Advocate General, also featured in the series as the office of the fictional foreign secretary.

Dover House has been considered by the team behind the making of Philomena, starring Dame Judi Dench, the true story of a woman who spent 50 years searching for the son she was forced to give up for adoption.

However, not all the actors who have visited Dover House in recent months have been making a film in the building.

Love Actually star and anti-phone hacking campaigner Hugh Grant visited recently for a meeting with Lord Wallace about the Coalition Government's response to the Leveson report.

The Scotland Office does not actively promote the building as a film set.

It is understood the filming in Dover House came about following word of mouth from other Whitehall departments.

Coalition aides say the money the building earns from doubling as a film set helps the Scotland Office pay for its running costs – as part of the Conservative-Liberal Democrat Coalition's attempts to reduce spending.

Whitehall sources stress most of the filming takes place at the weekends and care is taken to ensure it does not interfere with the day-to-day running of the office.

However, a question mark now hangs over Dover House's starring role in any future Hollywood blockbusters, at least in the short term – Deputy Prime Minister and Liberal Democrat leader Nick Clegg is moving in next year while work is being done to his office in Downing Street.

Sources suggest that, while his presence will not necessarily preclude filming inside the building, it could raise questions regarding security.

The Coalition is keen to allow its buildings and estates to be used as film sets, within reason, believing it is a good deal for the taxpayer.

Earlier this year the Scottish Government drew criticism for rejecting Dover House as its home during the Olympics.

Despite being offered the base for free, it instead spent hundreds of thousands of pounds to set up a separate base a mile away.

The Scottish Government spent £25,000 a day hiring out the Army and Navy Club in Pall Mall for the duration of the Games.

It was renamed Scotland House and played host to sports stars and dignitaries.

But Scottish ministers angered opposition parties after they revealed the exercise was estimated to have cost £400,000.

Dover House was built in the 1750s and was once the home of Frederick, Duke of York and Albany, son of George III.

It became the Scottish Office in 1885. Its most notable feature is an entrance hall in the form of a rotunda.