The criticism came after the latest arrivals, Graham and Olivia Uney, decided to leave, despite having only moved there in April.
The Uneys blamed the management of the island, owned by the National Trust for Scotland (NTS), and are now moving to Shetland.
The families say they are speaking out now because the charity has previously dismissed their concerns. They claim the departures from the Small Isles after the appointment of Stewart Connor as Canna's property manager two years ago are no coincidence.
Duncan and Alison Spence, who left in February for the South of France, criticised Mr Connor's role, despite the fact he is supported by other islanders.
Mr Spence said: "We left Canna in February after just seven months. I was the temporary gardener, but Stewart Connor decided my services weren't required. We had gone with a business plan which the NTS approved. I started doing rabbit shooting, but he terminated that. We had planned a craft shop and the NTS gave us a lease on a hut, but he put so many obstacles in the way."
Mr Spence claimed his manner "made our lives intolerable, as he has done to several others".
He said this was in marked contrast to the even more remote NTS island of Fair Isle, where people were happy to stay.
The first to leave last year were gardener Neil Baker and his school worker wife Deb, who had arrived in 2008 from west Wales with their two children, but now live on Islay.
Mrs Baker said the NTS could not give them the long-term security the family needed. She added: "We decided to stick it out and looked on Stewart's arrival as a positive development. We tried to be welcoming, we invited him to our house, but it was just rebuffed and we had no idea why. He seemed to pick on us."
Whenever they tried to raise concerns with him, she said, he swept them aside.
They claimed the NTS did not act over their concerns.
John Clare and Sheila Gunn arrived on Canna's shores in 2007 from Oban to run the guesthouse on the island. But they claim Mr Connor's behaviour forced them to leave for Eigg, costing them about £1500.
Ms Gunn said "We loved running the guesthouse. There was a problem about security of tenure, but Stewart Connor's conduct was the main reason we left when we did, walking away from a wonderful business. We didn't do that lightly. But he was obstructive at every turn."
Geoff and Eilidh Soe-Paing were there for more than five years with their four children. Mrs Soe-Paing's grandparents were from Canna.
Mr Soe-Paing, who has moved the family to Lochaber, said: "Stewart Connor's approach was definitely one of the main reasons we left."
Mr Connor is on holiday and could not be contacted, but Winnie MacKinnon, whose family have been on the island for generations, said: "The core of Canna's population is 100% behind Stewart Connor and supports the work he is trying to do. It is one thing having a go at the NTS, but singling out one person is out of order."
An NTS spokesman said: "Those who have previously chosen to give up tenancies or employment on the island are on record as having cited different motivations for their choices.
"It is therefore surprising that they have raised these new allegations so long after their departures."
He said the Spences' complaints had been thoroughly investigated, and while there were lessons to be learned, "we could find no evidence to substantiate the serious allegations made against the employee concerned".
The spokesman said the NTS always took complaints seriously and acted upon them appropriately, and added: "The trust is deeply concerned that allegations of improper conduct are being exposed in this way."
GONE: Canna settlers Olivia and Graham Uney.