Politicians and residents were already furious over the use of public money for the Forest Pitch, which was created to stage just two football matches as part of the 2012 Olympic celebrations.
Now, as the area is allowed to return to its natural use in the woods of the Duke of Buccleuch's Borders estate, teenagers have taken over the specially-erected wooden pavilion for underage drinking sessions.
Photos of youngsters drinking at the remote site near Selkirk have appeared on websites and infuriated parents.
One said: "I found out about my son going to the pitch to drink and grounded him.
"It is bad enough them drinking but what would happen if they became ill out there in the middle of nowhere. The hut should be pulled down immediately."
Political activist Ross Anderson from Selkirk added: "It was bad enough that public money was wasted on this daft project but now kids are heading there at weekends to get drunk.
"It is only a couple of miles from Selkirk and the village of Midlem is not too far away so it's ideal for the kids to do what they want without being disturbed."
Edinburgh-based artist Craig Coulthard, 31, was one of 12 artists who won commissions in 2009 to celebrate the Cultural Olympiad – the only one in Scotland.
He came up with the Forest Pitch project to stage two football matches, but Mr Anderson said it was a waste of money.
He said less than 100 spectators turned up on August 25 despite free transport laid on from Edinburgh and Glasgow.
Mr Anderson added: "I have spoken with the bus company who put on nine coaches, each costing £600, to run from Glasgow, Edinburgh and provide the local shuttles.
"They had 53 passengers in total for the whole day."
SNP Environment Minister Paul Wheelhouse said: "I realise the project was associated with the Cultural Olympiad and London 2012 Festival rather than being a sports project and had some positives such as involving Scottish schoolchildren in designing players' strips.
"However, as a representative of the south of Scotland and a Borders resident, I have some real concerns about the value in funding a project of this kind which appears to provide neither a local sports legacy nor a tangible artistic legacy, given the pitch will be allowed to return to its natural state.
"I very much regret it may now be a haunt for folk to indulge in underage drinking and therefore is now having a negative impact."
Mr Coulthard has always defended the project, and said: "I will be visiting the site in the next few days to check the situation. If there are problems with underage drinking or with litter we will not hesitate to contact the police."