The group wants the Government to fast-track plans to dual the carriageway from Perth to Inverness and is critical of proposals for average-speed cameras with a 40mph speed restriction for HGVs.
Reverend Alasdair Macleod, of Lochbroom and Coigach Free Church in Ullapool, was a civil engineer before studying to become a minister. He said: "Any idea that there are engineering or planning obstacles that would delay dualling the A9 until 2025 is disingenuous to say the least."
However, Mr Brown, who had invited the representatives to a meeting after they had written to him, said a number of sections on the road would be dualled well before 2025, with more than 50% to be complete by 2022.
He said: "The A9 passes through areas that are breathtaking and hugely important in terms of wildlife and landscape, not to mention people's homes. The suggestion we can somehow ignore these procedures and not consult people who stand to be affected by the upgrading work is not possible."
He said the Government was legally bound to examine the possible impacts of the programme on the environment.