'A faint memory emerged again and again – her father, a leather-bound fairy-tale book, a snow child alive in its pages.

She couldn't clearly recall the story or more than a few of its illustrations ... If there was such a book, could there be such a child? If an old man and woman conjured up a little girl out of the snow and wilderness, what would she be to them? A daughter? A ghost?"

In Eowyn Ivey's National Book Award-winning debut, The Snow Child, a middle-aged couple whose baby died long ago make a child-like figure out of snow. The next morning the husband catches a glimpse of a real child running away into the woods beyond, and their snow child is knocked down. Have the couple really conjured up an actual bodily child, in their loneliness and grief?

Fairy tales, modern and ancient, can conjure real-life little girls out of snow, just as they can conjure flying carpets, dragons, magical lands reached through wardrobes, princesses in towers and frogs that talk. But generally these are tales that are made for children. Ivey's novel, by contrast, is a fairy tale for adults, and it's a form that has become increasingly popular over the last couples of decades.

Angela Carter is the writer most often credited with bringing the fairy tale to adult attention, with her rewritings of Bluebeard and Red Riding Hood in her 1979 collection, The Bloody Chamber. Children's writer Neil Gaiman paid tribute to Carter five years ago, calling her "the first writer who took fairy tales seriously, in the sense of not trying to explain them or to make them less or to pin them dead on paper,"

In this, the 20th year since her death, she has just been named the "Best of the James Tait Black Awards" for her 1984 novel, Nights At The Circus. Carter's tales were warped and perverse, in the best way, with added feminist impetus – in her stories, heroines' mothers could shoot dead man-eating tigers on their own, and Red Riding Hood could laugh at the wolf who wanted to eat her.

But it's not just the reinvention of the fairy tale for an adult audience that has developed over this period. Its commercial and critical appeal has likewise grown massively, and often simultaneously. Today, Alaskan-based author Ivey's The Snow Child can reach the bestseller lists and be nominated for a Guardian First Fiction award, too, while UK debut author Jess Richards can be shortlisted for the Costa First Novel Award for Snake Ropes, which will "show you crows who become statues and sisters who get tangled in each other's hair, keys that talk and ghosts who demand to be buried ..."

These writers can also stay cool and hip, though, trading on their alternative image even while successful, such as Toronto writer Halli Villegas, whose The Hair Wreath makes children disappear and haunts adults with the ghosts of their pasts. Los Angeles-based Aimee Bender can seem almost counter-culture with her girls with hands made of fire and lovers who are sea-turtles, in her surreal fairy tales, yet she is also a bestseller. And "established" authors like Susan Fletcher and Alice Hoffman may weave elements of the fairy tale into their novels – where houses are called Glass Slippers and naked men emerge from the sea – and win awards, yet still manage to seem that little bit leftfield.

What is it that makes the fairy tale for adults both alternative and popular at the same time? How does the kind of story mainly beloved by children manage to perform this trick? In March this year, a new set of fairy tales by the brothers Grimm was found in Germany, but the month previously, a UK study found that one in five parents was rejecting Grimm's tales as too frightening for their children. This autumn, Philip Pullman published his own version of their finest tales, aimed primarily at adults. So what is too dark and perverse for children (despite the popularity of J K Rowling, Neil Gaiman and Pullman) would appear to be taken up only too eagerly by their parents for themselves.

Once upon a time, of course, that's the way it was supposed to be. As AS Byatt points out in her introduction to Maria Tatar's The Grimm Reader, the Grimms' tales were originally folk tales harvested by the brothers for adult readers. It was only when this collection failed to find popular and commercial appeal that they refashioned them for children. But fairy tales have always had their roots in an adult audience, as Marina Warner points out in From The Beast To The Blonde. Tales from the 17th and 18th centuries "solicited an adult audience; the older generation were being eased into taking pleasure in make-believe, in pretending they had become childlike again and had returned to the pleasures of their youth through tales of magic and enchantment and the homespun wisdom of the hearth."

This mixture of the childlike and the adult is perhaps what helps the fairy tale perform its double act of being both popular and counter-culture at the same time. But it could be argued that double-ness, or at the very least, contrariness, lies at the very heart of the fairy tale.

One of the Grimms' tales specially written for adults shows a "stubborn child" being buried by its mother after he becomes ill, but repeatedly sticking his hand out through the grave, until the mother herself comes along and swipes it hard and makes it retreat. Another story shows a mother planning to eat her daughters because she is starving; and in another, a mother-in-law tries to eat her grandchildren. These tales not only magnify adults' fears about the harm that may come to their children, they also magnify the part adults may play in their own children's harm. The fairy tale reflects "our secret fears and preoccupations" back at us, says Byatt.

And that reflection gives traditional fairy tales their edge, because it makes them reassuring on the one hand ("an imaginative world where... children escape the drab realities of everyday life") and terrifying on the other (the Grimms' tales "capture anxieties and fantasies that have deep roots in childhood experience").

For contemporary Scots fairy-tale writer Kirsty Logan, author of stories including The Coin-Operated Boy and The Rental Heart, that double element means that the fairy tale can allow us to talk about the un-talkable. Fairy tales now and in the past are full of forbidden relationships, class-crossovers and incestuous pairings. "There are lots of bad parents," she says, "father-daughter romances, betrayals by parents, abandonments, deaths. But they're metaphorical stories – almost every culture has a 'Cinderella' story because we all feel hard done-by at some point in our lives.

"'Cinderella' says that we'll show them in the end. People will always feel a need to understand things that happen in their lives and the fairy tale is one way of doing that. When someone dies, you want to be shown what to do."

If the fairy tale is where the taboo can reside more easily, the one place where we can safely explore our deepest darkest feelings, then the complexity of its morality holds firm, no matter how alternative or mainstream it becomes. It can modernise a Grimms Brothers' "original" in urban or "new world" settings, as award-winning Cassandra Parkin does in her New World Fairy Tales; or it can tap into local folklore, as Lucy Wood's recent collection Diving Belles does with Cornish tales. It can be both fantasy and reality, as Logan stresses, "a story with simple recognisable emotions, in a timeless setting with fantastical elements".

As Joan Acocella noted in a New Yorker article about the fairy tale earlier this year: "The reason that most people value fairy tales, I would say, is that they do not detain us with hope but simply validate what is. Even people who have never known hunger ... still have a sense of utter blackness, the erasure of safety and comfort and trust. Fairy tales tell us that such knowledge, or fear, is not fantastic but realistic."

By exploring what is still, or has always been, taboo in our culture, like a middle-aged couple's unvoiced longing for the child they have lost, it shows where relief can be found – in that curious mixture of the fantastical and the realistic that makes the fairy tale the irresistible, and even necessary, form it has become for us today.