The Honourable Alex Howard, heir apparent to the baronetcy of Strathcona and Mount Royal, owns the Colonsay Estate, as well as a home in Oxford.
The Gordonstoun-educated laird wrote an angry email to his council, which blamed locals for the disappearance of tons of gravel from a beach on his estate.
He alleged crofters had been removing estate property for generations and complained it was proving a "hard habit to break".
The claim has infuriated residents, who branded the comments offensive and said innocent people had been labelled "thieves" and "peasants".
Mr Howard has since been forced to admit the removal of gravel was the work of builders from the mainland - but he has not offered an apology to islanders.
The Colonsay Estate, which boasts one the finest collection of rhododendrons in Scotland, was passed to Mr Howard more than 20 years ago by his father, Donald Euan Palmer Howard, the 4th Baron Strathcona and Mount Royal, who still lives on the island.
The row began after Argyll and Bute Council threatened to take legal action against Mr Howard over the state of the Rubh aird ala ais beach.
Councillors said it was covered in large holes. Officials had investigated after a resident complained the beach had been left "looking like the Somme".
In an email released under Freedom of Information, Mr Howard explained gravel was being removed without his permission.
He wrote: "A number of crofters have extensively developed their crofts and I seldom see gravel being imported on to the island.
"It is extremely difficult to police this activity and many of the crofters have their own tractors, trailer and excavators.
"It is impossible to prevent those who have little or no regard for the rights of property - unless it is their own - especially as we have no police presence on the island."
One islander, who wished to remain anonymous, said the comments were very hurtful.
The resident said: "For generations people have taken gravel from the beach. We usually have to pay Alex about £100 for 10 tonnes. The same amount would be £1000 to bring in on the ferry. Technically it is his - but the stones are actually washed up by the sea.
"For him to brand us as thieves in an official letter to the council is going back 200 years. Does he really think he is the laird and we are just the peasants? People have grudged paying him for gravel over the years but it is much cheaper than importing it on the ferry.
"The system has worked well until some builders wrecked it by taking stones from really sensitive areas. But for him now to tell the world we are good for nothing thieves is very hurtful."
Mr Howard said: "I accept the person who took the gravel was a builder not from the island. However, they were working on a building that was being built on land that was just sold by a crofter.
"What's said in the letter still stands.
"I didn't mean to imply people had broken the law."