The party had tweeted the image of the former Labour Prime Minister and his wife Sarah, which made it look like they were exiting Downing Street, with the caption "all's well that ends well #LabourPlays".
The image of the Browns had been taken from a picture of the couple attending a memorial service to mark the 65th anniversary of the end of the war in Europe.
Conservative Central Headquarters (CCHQ) staff had been encouraging their followers on the social media network Twitter to share humorous versions of Shakespeare play names, such as Comedy of Errors, using the hashtag #LabourPlays, to celebrate the playwright's 450th birthday.
However, Labour demanded an apology and a Tory source admitted the tweet had been a mistake.
Labour MP Sheila Gilmore called its use "distasteful", adding that there had been "an appalling lack of judgement by Tory HQ" and urged the party to "make clear this kind of behaviour is not acceptable".
The Tory source said: "It was a mistake and inappropriate. We deleted the tweet as soon as we realised the origins of the picture."
The background image was first used in 2010 by The Sunday Times to illustrate a piece about the couple leaving 10 Downing Street.
l On Tuesday, Mr Brown, MP for Kirkcaldy and Cowdenbeath, visited Glasgow University, where he urged Scots to protect their pensions by voting No to independence.