The Task Force For the Future of Residential Care for Older People in Scotland was commissioned by ministers to look at how high-quality personalised care can be provided for the next 20 years.
While its report says there has been a general improvement in the quality of care, it says that cannot continue without change as it is "simply unaffordable".
Demographic changes are putting pressure on services, as highlighted in the Herald's NHS: Time for Action campaign, calling for a review of resources and pressures on the system as a result of the ageing population.
Fees for private providers are negotiated annually between local authority umbrella group Cosla and Scottish Care, which represents care homes, but the task force says: "Over the past few years these negotiations have run into significant difficulty … Many providers continue to operate at the margins of viability."
At present, the report says, local government spends £637 million a year buying care for older people and £111m on free personal and nursing care for those who pay for their own accommodation. However, it says, this needs to rise by 4-5% a year to be sustainable.
The group, made up of experts from the care-home industry, local government, the NHS, housing and charities, says ministers should also review the amount spent on free personal care as current rates are unrealistic.
The public dislike the current system which means anyone with property or capital worth £25,250 or more has to pay their full accommodation costs, it says.
Large numbers of people are above this limit when they first need a care home and families have to sell their homes to fund it.
The report says the Scottish Government should consider raising the limit to £118,000.
The taskforce says paying all staff the living wage would make work in care homes more desirable as a career option, but this would cost more. If funding is not increased, there is a greater risk of care-home firms going bust, it adds. The report also says more training in dementia is needed for the workforce and calls for an early-warning system to flag up struggling services.
Ranald Mair, chief executive of Scottish Care said the findings should be implemented without delay to avoid a crisis in the sector.
Mr Mair, a task-force member, said: "It has been clear for some time that more of the same is not the answer. Without proper funding care homes face an increased risk of going into administration and existing problems with staff recruitment and retention - which are already at crisis point in some areas - will worsen."
A Scottish Government spokesman said: "[The] recommendations will play a vital role in how we work with our partners to develop the Scottish residential care sector for the future."
Annette Bruton, chief executive of the Care Inspectorate, said the recommendations would be considered. "We are currently reviewing our entire inspection methodology, so the timing is very helpful," she added.