IN December 1949, a young Canadian doctoral student arrived on Unst, Shetland, the UK's most northerly inhabited island. His name was Erving Goffman, and over the next 18 months, he made a careful study of the local people, who at that time numbered about a thousand. What he saw, he never forgot.

For the islanders were notably standoffish. Visiting seamen learned not to look for more from them than a nod, or a brief conversation about the weather. But the locals were also shy of each other, and self-effacing to the point where the word "I" cropped up only rarely. No-one wept when the younger islanders left for the mainland. Eyes remained stubbornly dry even at funerals, and the local children sought to make themselves invisible as best they could.

Goffman, who went on to become a renowned sociologist, put the islanders’ shyness down to various factors, among them their Nordic ancestry – Nordic peoples were known for their shyness – and the crofters’ historical relationship with the lairds.

On Unst today, things have changed for the better, and according to Gordon Thomson, who chairs the community council, one transformative factor was the siting of an RAF radar base there during the 1950s. “The RAF personnel mixed in with the local community," he tells me. "Many of them met and married local girls. Maybe these men had more money and seemed more confident. Some of them brought their families with them and lived in married quarters; their kids attended the schools and mixed with local kids. Because of all of that, any shyness that there might have been on Unst – and there probably was some reserve – tended to melt away.” Television has also had an impact; and, in any event, "a lot of local people are quite chatty and outgoing".

This story – well-known in the Shetlands – is a reminder that, much like love, shyness is all around us. Shyness takes several forms: embarrassment, stage-fright, the state of being tongue-tied.

Many artists and writers have suffered from shyness. Angela Carter once wrote witheringly of her shy teenage self. The New Zealand writer, Janet Frame, was painfully shy and introspective as a child. Philip Larkin once told an interviewer that he was "extremely shy", adding: "Anyone who has stammered will know what agony it is, especially at school. It means you never take the lead in anything or do anything but try to efface yourself. I often wonder if I was shy because I stammered, or vice versa."

It can take a long time to grow out of your shyness. "I'm afraid my crippling shyness will ruin my career," was the headline above a reader's letter to a forum on a national newspaper recently. Read another: "My shyness and insecurity around women is still affecting me at 40.” Both of these writers were adults. Earlier this year, it was reported that the 30-year-old son of a London neurologist died after accidentally overdosing on a cocktail of legal highs which he bought over the internet in an attempt to combat his shyness.

A couple of years ago, broadcaster Paul Merton spoke about his own early shyness. Using the phone was extremely difficult. "It wasn’t something that made me sweat when I was doing it. But the contemplation of doing it, the anxiety of doing it ... I mean, what can go wrong with a phone call?” Author Joe Moran knows how Merton feels. When faced with a new phone number, he says, he cannot dial it "without having written down, like a call-centre worker with a corporate script, what I am going to say when the person I am ringing picks up".

Moran is professor of English and cultural history at Liverpool John Moores University. His latest book is Shrinking Violets: A Field Guide To Shyness. It's a fascinating read, not just for its roll-call of the legions of the shy (authors as diverse Nathaniel Hawthorne, Elizabeth Taylor and Agatha Christie; Charles de Gaulle, footballer Bobby Charlton, Dirk Bogarde, Morrissey, and computer pioneer Alan Turing) but also for its illuminating stories gathered from across the world, including that one from Unst.

Some of Moran's case-histories are truly affecting. Peanuts creator Charles Schulz was very shy at school and as a young man he was deeply self-conscious when trying to sell his cartoon strips to newspaper syndication representatives: his creations were drawn on extremely large panels that naturally drew attention to him. Even when he had found fame, he would be filled with terror at the thought of trips away from home. There would be times when his wife would drop him at the airport but he would immediately get in a taxi and head back home, sometimes arriving there before his wife did.

Charles Darwin described shyness as an "odd state of mind". It is indeed. Like many people I have known shyness; the very act of talking about it seems acutely self-centred and utterly paradoxical. There have been many times when I hesitated before walking into a crowded room. Even if its occupants were friends or colleagues, all of whom were minding their own business, it didn’t matter. Walking through the door seemed at that precise moment like stepping off the edge of a very high cliff – a queasy, clammy sensation that overwhelmed me, and rooted me to the spot.

On all of those occasions I was merely walking into a room, not addressing the occupants, which would frankly have been an unhinged nightmare.

I envy people who can address a crowd of strangers and make it look like the easiest thing in the world. How do they do it? On the morning after the Brexit vote I watched Donald Trump address a crowd of 300 journalists at his Turnberry golf course. Yes, the US presidential candidate has been addressing huge crowds for many years, but I was interested to see how he made it all look as natural as breathing when faced with expectant strangers equipped with TV cameras and microphones.

Mercifully, I have only ever addressed a crowd of people about three or four times in my life, and about all of them – well, the least said, the better. One of those occasions, unfortunately, was my wedding-day speech. The prospect of it hung over the day like a dark and malevolent cloud; the closer the appointed hour drew, the sweatier my palms became. I tried all sorts of mind-games to get round my fear of public speaking. Was it too late to dash out to the nearest newsagent and buy a self-help book? Well, yes, of course it was. Could I feign a sudden, calamitous bout of sickness? No, of course I couldn't. In the event, I remember, I stood up, my hands shaking, my gaze fixed maybe three yards in front of me, and croaked as many words as I could force out. That's all I can remember: I blanked out the actual details.

One last episode has just come to mind. Four decades ago, my family ventured south on holiday – Blackpool, possibly. We made friends with another family, as you tend to do when the weather is relentlessly grim that you have no choice but to seek indoor entertainment. Come the final day, we all assembled for a photograph on the front steps of the B&B. Almost all of us, that is. Finding the occasion just too much, I scuttled (le mot juste) back up to our bedroom, which happened to overlook the entrance. Through the open window, my dad's words drifted up: "Where is he?”

Shyness, says Moran, turns you into an onlooker, “a close reader of the signs and wonders of the social world”. He emphasises – and I found this especially interesting – that it is about much more than just timidity or fearfulness. “Shyness also arises from a kind of social deafness,” he says, “a tin ear for non-verbal cues, a sense that you have failed to grasp some invisible thread that holds communal life together.” Morrissey, who has long acted as a siren for shy teenagers, has written (in the Smiths song, Ask): "Shyness is nice, and/Shyness can stop you/From doing all the things in life/ You'd like to"; which is one way of putting it. Others have found that with shyness comes melancholy, and perhaps a slightly different way of looking at the world.

Perhaps it was inevitable that shyness, having long intrigued psychoanalysts and other experts, would attract the attention of the medical profession. In 2007 the American author Christopher Lane published a key book, Shyness: How Normal Behaviour Became A Sickness. “Shyness isn’t just shyness any more,” he wrote. “It’s a disease. It has a variety of overwrought names, including ‘social anxiety’ and ‘avoidant personality disorder’ … And since the early 1990s, when the US Food and Drug Administration agreed that powerful psychotropic drugs were suitable ways of treating these conditions, countless Americans and Britons have daily swallowed large doses of Paxil, Prozac, Zoloft and other pills for routine emotions that experts now consider medical conditions."

Moran, it emerges, once thought about asking for a Seroxat prescription, but declined. Seroxat is Paxil's British name; it has been marketed as a way of dealing with "social anxiety disorder" and has been used as an antidepressant. In the press, when it was first prescribed on the NHS, it was talked about (not entirely accurately) as a "shyness pill". Moran's reason for declining it, like the book itself, is illuminating: the sadness caused by shyness, he says, is real, "and helping others to take that edge off the sadness is a noble aim. But taking a drug for social anxiety – for feeling stupid, or bored, or unlikeable – feels like shouting at the wind, arguing with the rain. It feels like trying to find a cure for being alive."

Many strategies have been devised to help teachers work with shy pupils in the classroom. The Independent reported, in 2012, the success of one early intervention: after-school Pyramid clubs, which had been running for more than three decades. They "offer 10 weeks of carefully structured therapeutic group activities that encourage children to gain confidence and manage their thoughts and emotions ... Research ... shows that the clubs have a positive and lasting effect."

Online, there is no end of material that can help people address their shyness. The MoodJuice website, developed by Choose Life Falkirk and HNS Forth Valley's Adult Clinical Psychology Service, has valuable advice about shyness and social anxiety: it's a resource I wish had been around when I was younger.

If it's a possibility open to you, you can decide that your shyness is manageable. Mine doesn't affect my life to an excessive degree (as I grew older I probably became more impatient with some of my shy traits); and in those areas where it has an impact, I've learned to work round it. I can live with my shyness – for better or worse, it's part of who I am. I can, sort of, walk into a crowded room without feeling clammy. Just don't ask me to speak to an audience.

Do shy people have particular qualities? I don't know. Who's to say that any such qualities are not shared by people who have never known shyness? I came across an article online, in which the (shy) author says: "Shy people are known to be empathetic, sensitive, more given to altruism, more adept at behind-the-scenes solid work than the naturally confident and outgoing." It's an argument, I suppose, but at least one friend of mine is quite loud and assertive, and capable of empathy and sensitivity.

Moran sums the matter up so well, and puts into words the thoughts that I have never quite been able to articulate. We are shy, he says, because we know we are different from other living things. And because humans also carry a rare cargo of self-consciousness, we are uniquely aware that, for all our need for intimacy, we face the world alone. A little shyness around each other is surely forgiveable.

"I have fought all my life," he continues, "the sense that being shy is a personal affliction that has left me viewing life from its edges. This feeling was early acquired and now seems hard-wired, for no amount of mature reflection seems entirely to rid me of it."

Shyness may even be the master key that unlocks our understanding of humans who, after all, have a strange capacity for turning in and reflecting on themselves. Shyness, Moran says, "isn't what alienates me from the rest of herd-loving humankind: it's the common thread that links me to them".

Shrinking Violets: A Field Guide to Shyness (Profile Books, £14.99) is available now