Historic Scotland said the development would have "no significant impact" on the setting of Robert Owen's renowned model village, which has earned prestigious Unesco World Heritage status.
But conservationists are demanding the Scottish Government call in the planning application to South Lanarkshire Council as a matter of urgency after more than 2000 written objections.
John Campbell, QC, representing campaigners, said Historic Scotland's response to the application had been of "astonishingly poor quality".
He said: "Their assessment fails every known test of intellectual rigour which ought to be applied."
The proposed quarry will extend deep into the agreed "buffer zone" of the World Heritage Site and extract 3.6 million tonnes of sand and gravel over a six-year period. The development is being led by Cemex, a Mexican-based multinational.