A picture of Mobile Lovers was posted on the celebrated street artist's website on Monday and its location was quickly identified as Clement Street in Bristol.
Fans travelled to the street to admire the image, which was painted on a black piece of wood screwed on to a wall.
But at 4pm, a group of men from the financially struggling Broad Plain & Riverside Youth Project - situated just beside the piece - used crowbars to remove it.
Dennis Stinchcombe, 58, leader of the youth project, initially said it would be sold to raise funds to keep the centre open.
"I was approached by somebody who knows Banksy very well," Mr Stinchcombe said. "He's an artist himself and he said, 'you need to take that Dennis, get it into that club - it's what it is meant for'."
However, Bristol mayor George Ferguson asked for the work to be put back on the council-owned wall.
By 6pm, planks of wood had been screwed over the empty doorway, with a notice urging fans to visit the artwork at the club by paying a donation.
A smaller copy of the original has since been pinned up in its place.
The work is now expected to go on display in a museum while the question of its ownership is resolved.
Art fans from around the city and beyond flocked to see the piece after work yesterday, are were disappointed to find it had been removed.
David Bryant, 54, of Bristol, said: "I came all the way down here and I find it's gone and someone is making money out of it. That's not what Banksy is about, he is not about money."