The Scottish Campaign for a Fair Society (SCFS), Learning Disability Alliance Scotland (LDAS) and the Coalition of Carers in Scotland, have quit a working group set up by the local authorities umbrella body, Cosla.
They claim it has failed to tackle concerns about a postcode lottery of charging for services by councils, which they have dubbed a "care tax".
The three groups have joined 21 other organisations representing the disabled, older people and carers in tabling a Holyrood petition calling on the Government to end the fees.
The working group aimed to deliver greater consistency, and disability groups hoped it would look at who was eligible for charges and how much they pay. But the decision to leave means only two of the original five voluntary-sector participants are still involved.
Councils are allowed to charge disabled people, older people and people with long-term conditions a contribution towards the cost of help with daily tasks such as getting up and dressing, preparing meals or taking part in social activities. Nearly every local authority sets its own charges for the care it provides.
The costs can be higher if disabled people have an income, and payments such as disability benefits are taken into account in calculating charges, which can come to thousands of pounds a year.
Opponents say care charges have risen by 12 per cent on average while talks have been continuing, but differences in the amounts charged by councils across the country have increased.
Jim Elder Woodward of the SCFS said: "Nothing is progressing, variation is still commonplace, and disabled people are seeing their charges rising even as we continue to speak to Cosla. So we have decided to withdraw from their working party."
Mr Woodward said Aberdeen City Council had more than doubled its charging income from disabled people in the past two years, while West Dunbartonshire has more than trebled the amount it received in care charges.
The petition calling on the Government and local authorities to abolish charging has been backed by charities such as Sense Scotland, Quarriers and Capability Scotland. It says non-residential social care is essential for disabled people to achieve equality and human rights and would lift tens of thousands of disabled people out of poverty at a stroke.
Ian Hood of the LDAS said: "Every council in Scotland sets the threshold for paying care charges lower than the weekly poverty line of £177. Three-quarters of councils in Scotland say that disabled people with weekly incomes as low as £123 should be paying care charges.
He added: "Scotland is one of the richest countries in the world and there is no reason for disabled people to be living in poverty in our society."
Cosla's health and well-being spokesperson, Councillor Peter Johnston, said: "We will now actively seek to replace our former colleagues with other organisations that represent carers and disabled people. While we recognise that there are different views about whether to charge people for services, it is a crucial part of councils' income."