It suggested investment should instead be ploughed into the Scotland to London west and east coast rail routes.
The move by some of Britain's top bosses came after Labour said it could drop its support if costs rose above £50bn.
Alistair Darling, the former chancellor and transport secretary, also signalled his opposition. He said the costs of HS2, which some have put as high as £80bn, would drain resources from the rest of the network.
Simon Walker, Director General of the IoD, said the body's research suggested firms were not convinced by the economic case for HS2.
In terms of what the key priority should be, the survey showed 80% of IoD members felt investment in existing intercity rail services was most important, while 41% said HS2.
In 2011 a similar survey of IoD members found 54% rated HS2 as important to their business; this figure has now fallen to 41%.
Mr Walker said: "Businesses know value for money when they see it and our research shows they don't see it in the Government's case for HS2.
He said there appeared to be little enthusiasm on the whole, with almost every region expecting London to benefit most.
He added: "The IoD cannot support the Government's economic case for HS2 when so many of our members are doubtful of the benefits.
"The money would be better spent elsewhere, in a way that will benefit much more of the country. Investment in the West and East Coast main lines combined with a variety of other infrastructure projects would be a far more sensible option."
Mr Walker added: "It is time for the Government to look at a thousand smaller projects instead of falling for one grand folly."
At present, the HS2 plan involves a new rail link from London to Birmingham by 2026, to be continued to Manchester and Leeds by 2032. While an extension to Scotland has been mooted, no plans have been put forward.