It goes something like this: "It's been ruined" or "it's not what it used to be". It's almost as if they revel in disowning somewhere once cherished.
When I first went to Koh Samui on the Gulf of Thailand in 1991 it was frequented by backpackers who had grown tired of Goa, Bali or any other current day Nirvana.
The island's airport - with its thatched roofed arrivals hall and quaint little cars transporting you from aircraft to terminal - offered a handful of daily flights from the Thai capital on twin-prop aircraft.
There was a time going to Samui meant doing not much. Days consisted of lazing on unspoiled beaches and nights were spent striking up friendships. Chaweng was, and remains, the centre of life. Once a simple stretch of unmade road flanked by an adequate selection of bars and restaurants, it is now a heaving concentration of places to eat, drink, shop and be pampered.
It has also now acquired Starbucks, McDonalds, Tesco and Boots (yes, Boots).
Despite all that, Chaweng remains a place where you might, of an evening, stumble across someone you met earlier at a beach bar, at Muay Thai training or on a cookery course. And that is one of the enduring charms of Samui.
In the mid-1990s, if you rolled out of a bar at 2am you'd spend the next 10 minutes haggling with songthaew (shared taxi) drivers just to save a few baht to get back to your accommodation. Now, it's all metered taxis and terribly civilised.
Thailand's national brew, Singha beer, is less heady these days. It used to weigh in at 6.5%. Back in the years when I could take the pace I felt "embalmed" several times.
Today, there are corners of this island crammed with life, where pavement space is acquired only with cunning, where bars heave with bronzed holiday makers and where enterprising retailers and glamorous khatoey (ladyboys) will do everything within their power to command your attention.
And yet, you will still find superb places to eat and bars that wouldn't be out of place in chi-chi London or Manhattan.
Alongside the emergence of the stylish you will also find nightlife at its slightly less polished. It is these experiences that, for me, ensure Samui is still a special corner of this kingdom. Mario runs an Italian taverna off the main Chaweng drag. He's a kind of ageing hippy blended with Eurovision Song Contest entrant from around 1985. He has the look of someone who rolled up here 20 years ago and decided to stay put.
The surroundings are convivial and there's no doubting our host's eagerness to please. When the couple at the next table makes a slight observation of their main course he transforms into a sort of Gordon Ramsay on Valium.
In 1991 you didn't have places like the Samui Institute of Thai Culinary Arts where cookery courses are presided over by Leh, who trained as an electrical engineer ("too boring") before taking up the wok.
Leh has that love of sanuk (fun) beloved of Thais. As she looms over a table strewn with exotic ingredients she instructs us in the medicinal as well as culinary benefits of everything from galangal to tamarind.
"Lemongrass. Have vitamin C, very good for skin, teeth," and looking to my shaved head, "good for hair," which she delivers with a slight smile that then develops into a full-blown guttural laugh.
We prepare ingredients, the 12 amateur cooks pummelling away at a pestle and mortar before being taken to a kitchen in which woks are assembled in a semi-circle. This is a fun way to spend a few hours. Leh and her team are great hosts.
You wouldn't have found somewhere like Tamarind Springs back in the early 1990s either. Located on Samui's east coast within a tropical forest where lush coconut groves blend with huge granite boulders, this has been named "spa of the year" by the Thai tourist industry - and it's easy to see why.
An afternoon here comes with a price-tag but the beautiful gardens, secluded corners and seductive treatments will send you away rejuvenated.
Samui has changed. But if you're savvy enough to look behind the concessions to the 21st century there remains a soul and integrity to this island that is quintessentially Thai.
I'm smitten all over again.
Selective Asia (selectiveasia.com; 01273 670001) offers a nine-night trip including five nights at the luxury Anantara Lawana Resort and Spa on Koh Samui combined with four nights in the north of Thailand - including two at the Anantara Golden Triangle Elephant Camp and Resort with an elephant and mahout training day included - from £1523 per person including private transfers, internal fights, breakfasts and some other meals. International flights are not included but can be booked through Selective Asia.