There is a house.

There is always a house. Sometimes it sits in the country out on its own, sometimes on a causeway where the tide cuts it off twice a day. Sometimes it just sits at the end of your street, empty and unloved. From time to time you'll pass by, and you always look. What is it about it that unnerves you?

There are stories told about this house. Kids tell them on summer nights: silly, gory stories made up to frighten the new kids in the street. There are grown-up versions too, whispered over late-night drinks at house parties "… And the child was never found …" You come home and tell them to your partner and the pair of you laugh.

Still.

Still, on dark nights when you take the dog for a walk, you tend to go the other way - though even then you always end up passing it on the way home. It is silent, decaying; a rotten tooth of a building. The house seems to list. Subsidence, you imagine. The windows are dark holes, like empty eye sockets. The white paint on the frame is battered and flaking. In one corner the frame is even cracked. You can see a shard of glass poking out like a bone through skin. You tend to walk past quickly. Only when you're back home do you realise you've been holding your breath.

Maybe it's because of that one night. That night the dog, chasing a scent, pulled you right instead of left and, not thinking, you followed him. Right past the front of the house. And you looked up and you thought you saw … something. A smear of white in the highest room. Probably just an old bit of curtain, holed and tattered, blown in front of the window for a moment. Must have been. The vision stays with you for days, though. Maybe it's because you can't shake the idea that when you first saw it you thought it was a face.

BACK at the turn of the 20th century, Montague Rhodes James, the Dean of King's College Cambridge, would gather colleagues, friends and choirboys together after the Christmas Service and entertain them, by candlelight, with the ghost stories he had written.

More than 100 years on, electricity may have replaced the candles and television the human storyteller, but the tradition remains. Next Saturday, Channel 5 is screening The Haunting Of Radcliffe House, an original ghost story starring Olivia Williams and Matthew Modine. And on January 1, The Woman In Black: Angel Of Death, a sequel to the hugely successful film adaptation of Susan Hill's Victorian pastiche, is released in cinemas. Even now we want to frighten each other in the dark.

The Christmas ghost story has long been one of our seasonal traditions, as predictable as Brussels sprouts and terrible Christmas cracker jokes. We associate it with Charles Dickens and the Victorian era - over the next week it will be difficult to avoid Dickens's A Christmas Carol. That is partly a legacy of publishing history. As Michael Sims explains in his introduction to his recent anthology of Victorian ghost stories, titled The Phantom Coach, the repeal of the newspaper stamp - the excise duty on paper - and the advertisement duty in the 1850s led to an explosion of magazines and periodicals, and a concomitant surge in the demand for short fiction to fill them. The ghost story was just one of the genres that benefited. The subsequent success of A Christmas Carol, no doubt, did much to establish the link between the season and the spook story.

"A lot of our Christmas traditions do date back to the Victorians," says Scottish writer Louise Welsh. "It's the same with the tree and Christmas puddings, and even the idea of the nuclear family."

At times, that connection between the Victorians and the ghost story hardens into an orthodoxy. The appeal of Susan Hill's 1983 novella The Woman In Black is surely in the way it cleaves so closely to the familiar model: the isolated house, the screams in the mist, the appearance of the vengeful ghost. But what was contemporary to the Victorians and Edwardians is now period detail.

Still, there's nothing intrinsically 19th century about the form. Welsh is currently putting together a host of ghost stories for publication in a collection next year. Her selected authors stretch from Pliny the Younger to the author Sarah Hall, who, Welsh points out, was born in the 1970s. The ghost story is one of the oldest story forms. You can find ghost stories in the Bible if you look, and they exist in every culture in every time. "When I think of our ancestors, those ancient people sitting around on Skara Brae at night, the wind whooshing around the house, I imagine them telling ghost stories," says Welsh. And when else would they gather but at the height of winter, on the coldest, darkest nights, enjoying a comfortable shiver wrapped indoors safe from the feral night.

"And still in the 21st century, we're telling similar stories to ourselves and that might be through the oral tradition or it might be through literature or television or computer games or whatever. We're still frightened by these same stories, by what's around the corner, what's in the dark."

The death of God and the rise of evolutionary theory hasn't killed this urge. We may be a more secular society in Western Europe than a century or two ago, but the fear of the dark remains. "I defy anyone," Welsh says, "to be out walking in the dark in the Borders or Ayrshire or up on the islands and your car's stopped and you don't have a torch and it's a wild and windy night and you have to get out and walk and not have a wee shiver."

The question is, what are we scared of?

That "face" at the window stays with you. It's as if it looms out at you every time you look into the shadows. You try to find out about the house. You Google the address, ask your neighbours. But no-one seems to know anything concrete. Does it really have a tragic history? If so, no-one can tell you when it was meant to have happened or anything about the people it happened to. All the stories, like the house itself, seem empty when you look into them.

But still there's something about that place. You feel it every time you pass. A wrongness. "The only thing that's wrong with it is that it's lowering the value of ours," your partner says, picking up the car keys to head into town for some last-minute Christmas shopping.

You don't have to look too hard to spot the ghost of Sigmund Freud at work in horror fiction. In his introduction to the 2013 Oxford World Classics edition of MR James's Collected Ghost Stories, Darryl Jones cites the story Casting The Runes, in which a Mr Dunning, lying in bed in the dark, gropes for matches under his pillow. The matches aren't there. But something else is, "a mouth, with teeth, and hair about it … not the mouth of a human being".

For Jones this is clearly an image of sexual terror. In it he sees a glimpse of James's homosexuality and his fear of women. The "mouth" is a symbol of the myth of the vagina dentata, he suggests.

How often does the Victorian and Edwardian ghost story deal with the return of the repressed? How many wronged, vengeful women haunt its pages? And how often does a native curse blight the life of an Englishman? In Rudyard Kipling's Mark Of The Beast, when a leper casts a spell on an Englishman who has blasphemed a native temple, it is difficult to think of this vision of empire being an example of the return of the imperially repressed.

As the HP Lovecraft scholar ST Joshi once said, ghosts are "the objectification of a psychological state". In that case, God can be dead, alive or on holiday in Dunoon. It doesn't matter. It's not the afterlife that is bothering us, it's the life we're living that we're scared of - or the one we're imagining.

Yet there are other, more obvious, more eternal fears at play here too. This year's great celluloid ghost story, Jennifer Kent's The Babadook, a story of bad dreams and a cursed book, has at its heart that most primal of human emotions: grief. When we gather together at Christmas there will be a moment when we remember those who are no longer there; lost fathers or mothers or grandparents or brothers or sisters. Aren't they, inevitably, the ghosts at every Christmas feast?

You come home from walking the dog, lie down on the sofa, feel your eyes begin to close … You wake to find yourself outside, walking towards the house. The front door is open, the darkness behind it is inky black and seems to spill out towards you. But that doesn't stop you. You step through the door and go inside.

The house is a mess, of course. When your eyes adjust all you can see is dust and dirt, mottled mirrors, broken toys, seats half-eaten away with damp and mould and rot. You drift further into the building, climbing stairs that squeak in protest at every step, higher and higher until you reach that top room. You pause outside. Do you hear someone whisper within?

You enter. And find … nothing. There is no curtain near the window, no furniture, no bed. You walk over to the window and look out. Through the smeared, spiderwebbed dirt you look down and see someone looking up. It's only after you wake up that you realise you were looking down on yourself.

Ghosts, then, are window dressing. They're an externalisation of our fears. And sometimes we externalise that in bricks and mortar. The haunted house: horror fiction is full of them. "Hill House, not sane, stood by itself against its hills, holding darkness within," Shirley Jackson writes on the first page of her novel The Haunting Of Hill House. It is most definitely an example of what Stephen King once summarised as "the bad place". A depository of the things we call evil.

You can find similar buildings in Sarah Waters's The Little Stranger and Mark Z Danielewski's meta-horror novel, House Of Leaves. They can be places where past crimes are constantly replayed by ghosts or by its possessed occupants. The Overlook Hotel in Stephen King's novel The Shining - and in Stanley Kubrick's film version - may be the most celebrated modern example. It is as if the walls of the hotel had soaked up the horrors of the past and were replaying them to us in the present. (The creator of the Quatermass series, Nigel Kneale, wrote a teleplay called The Stone Tape in 1972 based on this very idea). It is an idea that plays out in real life too. "Would you buy a house where a horrible murder took place?" asks Welsh. "We still flatten these places."

But what are we watching when we watch Kubrick's version of The Shining? For the writer Paul Mayersberg, reviewing it for the film magazine Sight & Sound back in the winter of 1980, "the central horror of The Shining is family life. For a child there can be few characters more frightening than his angry father. Danny, despite his stoicism, is terrorised by his father. Wendy is terrorised by her violent husband. Jack is frustrated to the point of rejection and violent aggression towards his family".

And at some level this is what all these stories are about: violence, neglect, destruction, pity. Ghost stories are about moments in which someone has failed. And, of course, they are about failing bodies. We can't escape the idea of death, can we? The inevitable terminus. We have journeyed from a fear of the unknown through a fear of our unknowable psyches to a fear of the very architecture of the self. What are we frightened of? Ourselves, of course. The messy reality of our material being, the home we all live in. And the knowledge of what will inevitably happen to it.

When you eventually wake you feel dazed. It's like you can't shake the dream; as if it is somewhere inside you, locked away in a room that you can't get to but know is there. You feel … you just feel wrong.

You stagger to the bathroom, turn on the light and look in the mirror. What you see shocks you. You don't even recognise yourself. Your eyes are bloodshot black. They look like empty windows. Your mouth is an empty hole, like a door that's been kicked in. The left side of your face from the eye socket down looks to have listed and slipped.

It is not your face. Or rather, it has changed. For a moment you can't think what it reminds you of. Then you look out of the window towards the end of the street and you know. You turn back to the mirror. The haunted house that is now your face is looking back at you.

We tell ghost stories because there's a pleasure in being scared. "It's nice to choose to be frightened, when one need not be," author Elizabeth Bowen once said.

We tell ghost stories because they allow us to talk about things that may be too painful to talk about. And we tell ghost stories because we know, much as we may like to pretend we don't, that we are all ghosts. Or will be soon enough.