Those who work in the sector expect a small rise in the number of people with learning disabilities of around 1% each year.
But, according to new figures, the number known to Scottish local authorities dropped in 2010-2011 by 4.6%, or 1355 people. This follows a smaller fall the previous year.
Ian Hood, coordinator of the Learning Disability Alliance Scotland (LDAS), said: "There are no fewer people with learning disabilities in Scotland than a year ago.
"These statistics tell us that fewer people are getting the help they need from councils."
The Scottish Consortium for Learning Disability's eSAY project was set up in 2003 and is the Government's main source of information about the number of people with learning disabilities or autism.
It found there were 26,036 such individuals known to councils in 2011, down from 27,391 in 2010, and 27,671 in 2009.
The figures show 14 councils had reported a reduction in numbers last year, with some seeing big falls. They include Glasgow, where 135 fewer people with learning disabilities were recorded, North Lanarkshire (862), Edinburgh (311) and Aberdeen (155).
While some of the reduction may be due to more accurate counting – several councils have taken steps to reduce double-counting of people using more than one service – LDAS said the closure or withdrawal of council services accounted for the bulk of the change.
LDAS is an umbrella group representing leading learning disability charities including the RNIB, Quarriers, CrossReach, Down's Syndrome Scotland and Enable Scotland.
Mr Hood added: "We are seeing the consequences of the first stages of public-sector spending cuts. For years we have been told there was going to be a big increase in the numbers of people with learning disabilities and that resources had to be shared out better.
"It turns out that people are getting less support all over the country."
Mr Hood said many people were now being told they had to have "substantial or critical needs" before they could be offered services.
This has two effects on the statistics, he claims: existing users of services are seeing them cut or reduced, and new cases need to meet a higher threshold before being offered help such as daycare.
Mr Hood said the reduction in official numbers was not unique to Scotland. He said: "Mencap, the English charity for people with learning disabilities, found in a recent study that 28% of English councils had seen a similar process of reduction of adults known to local authorities.
"In Scotland, nearly 44% of councils reported a reduction. And it is likely similar reasons apply here too, with the number of case closures coupled with the increasing difficulty to get accepted as eligible for social care services leading to this fall in numbers."
Mr Hood said the Scottish Government should seek an explanation from councils.
He said: "People are being pushed out of the back door, and that is showing up in the statistics 18 months later.
"If we are deciding people with learning disabilities no longer need services, that shouldn't be done by the back door through a process of 'modernising' day services. We need to have a public discussion about this."
Peter Scott, chief executive of Enable Scotland, said the statistics also revealed a worrying number of adults with learning difficulties who were aged 50-64, but reliant on a parent for their care. There were 542 such cases. "Many lifelong carers tell us they want 'peace of mind', in particular, knowing what will happen to their son and daughter when they are no longer able to care. We hope to see further progress made in helping to support these families to plan for the future," he said.
A Cosla spokesman said it was up to councils to allocate services and set eligibility criteria: "This is a budgetary decision for individual local authorities based on their own areas and circumstance," he said.
Contextual targeting label: