There's a peculiar irony in interviewing the quiet man of English letters in a London restaurant filled with noise.

Loud laughter, the clatter of crockery and the hiss and bang of a huge coffee-making apparatus make for a cacophonous recording when I listen again to my three-hour lunchtime talk with Graham Swift.

I have to strain to hear everything the Booker prizewinner says - as I did over lunch - since he's so soft-spoken, so enigmatic, so unshowy, despite the boldness of his magnificent body of work, to which he's about to add a fine collection of new short stories, England And Other Stories, which he'll be discussing at the Edinburgh International Book Festival.

Unlike more gregarious authors of his generation - he was 65 last month and in 1983 was named alongside Martin Amis, Julian Barnes, William Boyd, Ian McEwan and Salman Rushdie as one of Granta's 20 best young British novelists - he's a private man. He does not spew out details of his life in his work, in interviews or on social media; indeed, he rarely switches on his mobile phone.

"I am certainly not a writer who writes autobiographically," he says. "I don't write from my own experience." The closest he has got to writing a memoir is one work of non-fiction, Making An Elephant (2009), a superb collection of new and old essays, in which he interviews or is interviewed by writer friends, such as Patrick McGrath and Kazuo Ishiguro.

"Making An Elephant is the exception that proves the rule about not writing autobiographically," notes Swift. In this beautifully written, self-deprecating book, he reveals some of his poetry, ponders over his early career, tells how his two best-known, award-winning books - Waterland and Last Orders - were "adapted" by filmmakers, and writes lovingly about his father, who died in 1992. The elephant of the title was a wooden toy Swift made for him.

His south London childhood, with older brother Peter, was "sunny and secure". His father, a wartime fighter pilot, became a clerk in the National Debt Office; his mother was "a great bringer-up of children. The myth that the unhappy childhood makes a writer doesn't apply in my case. I realised later that my parents were struggling to make ends meet but there was no sense of shadows. However, I wanted to do the individual thing, to break free, to write, to have adventures. My parents gave me cosiness but not adventure, that had to come from me."

He remains, though, as someone once remarked, a closed book. "Oh, I don't think I like that," he responds, his lean, craggy features a picture of amused puzzlement. "If I am a closed book, I'll take that in a kind way. Making An Elephant opens me out if you want to know a bit about me, the writer, because the me in that book is very much me the writer. I have a life; I definitely do have a life to live and it's a very nice life. I'm extremely lucky, very, very fortunate to have this life which has enabled me to travel all over the world. Maybe it's not a particularly exciting life but then, in a real sense, writing is my life."

And anyone who loves literary fiction is grateful for that. For Swift is surely one of our foremost novelists, a master craftsman, often described as England's laureate of the everyday, with his wonderful novels, from his debut, The Sweet-Shop Owner (1980) and Last Orders, awarded the Booker Prize in 1996, to his most recent, Wish You Were Here (2011). He finds the extraordinary in ordinary lives while considering the bigger issues of life - death, birth, marriage, sex - always revealing the hidden poetry in men's lives. His protagonists are often male - window cleaners, barbers, coastguards, doctors - although there are several stories in the new collection told from the female viewpoint, particularly Yorkshire, an unforgettable, dark story about a 71-year-old woman's terror-filled, sleepless night.

Educated at Dulwich College, Cambridge and York universities, Swift has written nine novels, which have been translated into 30 languages, and an early collection of short stories, Learning To Swim (1982). "It's enough novels to make me forget how many there are," he says, counting them off on his fingers. England And Other Stories is his twelfth book - there are 25 stories in it as opposed to 13 in Learning To Swim.

"I began by writing short stories," he says. "I was always happy writing them; I didn't see them as a route to writing novels. I'd have been quite happy just writing stories because I think the short story is as great a form as the novel. They are not in contention. But one day I found myself writing a novel, The Sweet-Shop Owner, then another and another, and I was a novelist.

"Whenever I finished a novel, I was always ready to go back to short stories but they didn't come to me. Until I started England And Other Stories I hadn't written a short story for decades. Then suddenly I was writing story after story. It was almost as if I was writing one and there were a couple more waiting. I honestly can't say what a joy it's been to get back to the form. It's been exciting because it's not as though I had to learn how to do it again. It's almost as if the short story writer in me had been dormant and had then been raised again."

He estimates that he threw away a third of the stories he wrote while working on the collection, which includes stories that make you laugh, such as Ajax, in which a professor of Greek recalls the "weirdo" neighbour of his childhood, and others that pierce the heart - Fusilli, telling of a grieving parent doing the weekly shop, while recalling his soldier son's last phone call from Helmand.

"The great thing about writing short stories is, if you realise one isn't working, it's much easier to ditch it," Swift says. "If you're on page 75 of a novel and it's not working, then that's painful. In the past, I've thrown a whole novel into the dustbin." After writing Last Orders it was almost seven years before he published another novel, The Light Of Day (2003). "It didn't mean I wasn't writing stuff all the time," he says. "I was, but it didn't bear fruit that I liked. In any case, I don't feel that if a day goes by and I haven't put something on paper, it's a bad day."

Nonetheless, he's still writing short stories and is working on another collection. "I don't feel like a novel at the moment, although I think the evidence is that I'll probably write one. I felt early on that I was writing these stories for a book. Not just a book but some kind of integral, single enterprise. I'd written two when I realised that what was happening was what the title of the book says, England And Other Stories. Then I found myself writing the story called England, which ends the book. All the stories are about some not definable but collective thing - let us call it England. Pieces of England but different pieces, different Englands, because England is a mixture.

"I am very, very English but I also feel European and I think that's because I often find myself talking about my work in Germany or France, say, and, of course, Scotland. I feel I'm less of an isolated Englishman than I might have been because of my writing. This book is and isn't about England. It's about the territory in here," he says, touching his chest.

"The body is the country we all inhabit, it transcends nationality. There's a lot about bodies in this collection and a lot about the heart, a sprinkling of sex, too, and a great deal of flesh and blood since war, from the Civil War, when England was at war with itself, to the First World War and then Afghanistan, is the backdrop to several stories. I hope, though, that the title is provocative but engaging in the way that a book called Scotland And Other Stories would make me sit up. It's also an irony - it's England, it's just us. But are we so different on either side of the border?"

It is an interesting moment to be bringing the book to Scotland on the eve of the referendum, he remarks. "I wonder what are the British Isles going to be like in three months' time? It's not my business, though. It's between Scotland and Scotland."

When he comes to Edinburgh, he predicts someone will ask him the inevitable question. "Where do you get your ideas?" He sighs: "People think I'm being evasive but honestly all my work begins with nothing, it's almost as if it's some sort of tingle. It's like when you wake and try to catch a dream. I start with something so insubstantial that I'm always astonished that this next-to-nothing thing turns into sentences and paragraphs and pages."

For 30 years Swift and Rodd - whom he married on February 29 in 2000, "since it makes for interesting anniversaries," after having been together since meeting at York University - have lived in Wandsworth. He has written all of his books in the same room in their house, sitting down at his desk at 5.30am and working until lunchtime. "Then," he says, with a puckish grin, "I am free to go out and play. Which usually involves a lovely lunch like today's - and lots of talking," he says.

So is his study filled with the ghosts of all his characters? "No, not ghosts exactly but, oddly enough, I do feel that all my characters are out there in the world, that I might meet them at any moment and that I'd know them." It is, of course, a feeling shared by his readers, that the people he writes about with such empathy are all around us - in England or Scotland.

When I say this, he responds: "It is extraordinary that something produced in isolation by a writer sitting down in a room is received in isolation and shared. It's that communion, there's nothing like it. It's a really wonderful, democratic, humane thing. Nothing beats the magic of storytelling."

England And Other Stories by Graham Swift is published by Simon & Schuster, £16.99. Graham Swift will be in conversation with The Herald's Literary Editor, Rosemary Goring, at the Edinburgh International Book Festival on August 12.