Letters have been sent to Health Secretary Alex Neil highlighting the problems on Mull, which have been brought to a head with the decision of the doctors in Tobermory to opt out of 24-hour cover.
It is claimed this is directly contrary to the undertaking Dr Claire Lavery and Dr David Chapman had given to their predecessors when they arrived to take over the practice under two years ago.
In the letter to Mr Neil it is claimed the approach of the doctors, who are a couple with a family, is now causing fear and concern for residents and visitors, as well as among the other healthcare workers on the island.
It says: "They have failed to engage with the island ethos of cooperation and harmony and have alienated themselves from their patients and most of the other healthcare workers on the island."
Residents also allege the GPs have also been responsible for the dismissal of a practice manager who had served the practice without complaint for 17 years. She is now understood to be in the middle of a protracted unfair dismissal claim.
The letters to Mr Neil were sent after a public meeting was called to discuss the issues, which more than 100 attended.
One who attended was councillor Mary-Jean Devon, who lives in Tobermory. She said: "The turnout at that meeting speaks volumes.
"Everybody knows doctors have a very difficult and challenging job, and have to be able to live their own lives.
"But people on the island are now worried because they see the committed service they enjoyed for as long as anyone can remember, and the trust built up by previous GPs, disappearing. I can't remember a time when health was such an issue on the island."
She said the announcement on ending out of hours cover, which would now have to be undertaken by locums, was the latest setback.
It follows news that the two GPs who cover the Ross of Mull will leave at the end of next month, while one of the two in the Salen practice is also going. This is out of the island's total of six general practitioners.
The islands of Iona, Ulva and Gometra also depend on the Mull doctors.
Mile Shilson, chairman of the Mull and Iona Community Council, said there was also a long-standing worry that while there are two ambulances on the island and two crews, only one crew is on at any given time.
"This means it can take it two hours to get from one end of the island to the other."
Neither Dr Lavery or Dr Chapman wanted to comment on the allegations.
However, an NHS Highland spokesman said the health board had met representatives from the community to discuss GP services on the island and had arranged a number of drop-in sessions for people to talk to NHS Highland, NHS 24, the Scottish Ambulance Service and others.
"We are also in the process of setting up a Project Group that will be looking into the provision of a sustainable in-hours and out-of-hours GP and community hospital service that meets the needs of the community, he said, adding that local people would be represented on this group."
A spokesman for the Scottish Government said it was working with NHS Highland on Mull. "We know some health boards face significant difficulties recruiting in remote and rural areas of Scotland, including Mull," he said.