Saturday morning’s breakfast was ruined by a voice on Radio Four disparaging Edinburghers as “neurotic”. This commentator also claimed that the bulk of tourists come to the capital during the Edinburgh Festival, drawn not by the architectural charms of one of the most beautiful cities in the world but by its stand-up comedians.

Barely had we brushed the spluttered toast crumbs from our jerseys than emails were flying, one suggesting that the speaker, Richard Williams, Professor of Contemporary Visual Cultures at the University of Edinburgh, might soon have a career as a stand-up himself. Another asked why, since comedians could equally choose Cumbernauld or Glenrothes in which to perform, they instead elect to come to “dowdy old Edinburgh”.

Williams was commenting on the news that Unesco is sending representatives to report on Edinburgh City Council’s recent planning decisions, to reassess its status as a World Heritage Centre. This unusual step has been prompted by several ugly and incongruous building developments that threaten the aesthetic heritage of the city, about which I railed in this column last month. The so-called Walnut Whip hotel and retail complex that is to replace the hideous St James Centre; proposals to turn the Royal High School into a luxury hotel; and the Caltongate, a slick edifice of glass and massive stone in the old town, are but the most egregious examples of a council blind and deaf to the city in its care.

Asked what all the fuss was about, Professor Williams replied that “Edinburgh being Edinburgh, any sort of change does produce a rather neurotic reaction.” The word splenetic would better describe the response to his comments, which were not just patronising, but offensively glib. While the citizens of Edinburgh have their faults, neurosis is not one of them. Sang froid is their default position, along with a certain coolness of demeanour. Only such vandalism as the council has sanctioned of late could raise the mood from its arctic norm to a temperature approaching molten. But the council is not the only villain. Few have done more to rip the heart out of the city than the University of Edinburgh itself, whose demolition in the 1960s of one of the most perfect civic squares in Europe to make way for dreary tower blocks and a joyless library was an insult to the idea of higher education.

Given the part his employer has played in diminishing the city, Williams ought to have been more sensitive. What he and others fail to realise is that if Edinburgh’s architecture is further compromised, then much that makes it such an alluring, unique, enchanting place, will be forever lost. People don’t come here primarily for the climate, or the calibre of its arts, comics, or cuisine. Like the founders of the Edinburgh Festival, they first choose it because of the kind of city it is: a remarkable fusion of rugged medieval and elegant Georgian, whose split personality and salty sea air reflect the temperament of the country it serves. Almost all of its other assets have evolved from the appearance and nature of the place. Whoever thinks that buildings and streets, vistas and skylines are irrelevant should try living in Leith’s banana flats for a month.

As those benighted high-rises, and many other contemporary carbuncles show, serious mistakes have already been made. Princes Street is a monument to the soul-destroying brutalism of the sixties and seventies, its once sternly impressive frontages a cacophony of cement and plate glass. Meanwhile, housing estates on the outskirts are less well designed and more conducive to misery than many of Her Majesty’s prisons. As for the unctuous pandering to the hospitality industry, the sight of tented annexes for restaurants and bars in the middle of the New Town is heart-breaking. Indeed, they are so unsightly one would think they had been sanctioned not by our elected officials but by some ill-wisher, hell-bent on undermining Edinburgh’s credentials as anything but a venue for shopping and bingeing.

The prospect of losing its Unesco badge does not alarm critics appalled at the council’s planning decisions. This relatively meaningless accolade is of more interest to the tourist trade than to those eager to preserve Edinburgh’s character. These naysayers, however, are wrongly described not just as neurotic but as rear-guard, reactionary, and afraid of anything modern. Yet we are not at all averse to doing things differently, or embracing the new. Quite the opposite. For a start, we’d like a more sophisticated, intelligent, responsible and aesthetically educated council. We seek greater acknowledgement, among politicians and educators, of the great benefits a harmonious urban landscape brings to all who live and work there. Above all, we want Edinburgh and its history, as written in stone, to be given the respect and honour it deserves. Far from being frightened of change, we are hungry for it.