He's hidden behind a broadsheet newspaper. The photograph is captioned: "Portrait of the author." A second photograph shows him face-down, newspaper discarded: "The author's opinion of the Sunday newspaper."

These two images by Therese Mitchell, the photographer wife of "the author" Joseph Mitchell, bookend a collection of his early journalism, My Ears Are Bent. The fact that "the author" remains anonymous is symbolic of the life and work of this brilliant reporter, surely the finest America has produced and who once wrote: "A newspaper can have no greater nuisance than a reporter who is always trying to write literature." Ironically, the most subtle of stylists, he did write literature – great literature.

Mitchell was born in 1908, his father a North Carolina tobacco farmer, his mother of Scottish ancestry. When he was growing up, he recalled summer meetings with his aunts' families in a picnic grove "in back of an old church, out in the country". It was an old Iona Church, "a Scottish Presbyterian church that my mother's ancestors had helped build". He moved to New York City when he was 21, becoming a newspaper reporter and feature writer until 1938 when he joined the New Yorker, where he became perhaps the best single writer that magazine has ever employed.

If that sounds biased, I hero-worship Mitchell, the fedora-wearing "poet-laureate of old New York City". For years I have traced his beat from Greenwich Village to the Staten Island Ferry and the Bowery, from the Grand Central Oyster Bar to the Fulton Fish Market (now relocated to Hunts Point in the Bronx), always accompanied by a tattered copy of one of his books – either his masterpiece, Joe Gould's Secret, or My Ears Are Bent, or Up In The Old Hotel, re-published this month in paperback. Of course, I am not alone in my regard for Mitchell. His many devotees include Martin Amis, who claims: "If Borges had been a New Yorker, he might have come up with something like Joe Gould's Secret."

The last piece of writing Mitchell ever published, this essay-novella first appeared in the New Yorker in September 1964. He had first written about Joe Gould, "a blithe and emaciated little man who has been a notable in the cafeterias, diners and barrooms and dumps of Greenwich Village for a quarter of a century", in 1942. Gould told Mitchell about a work in progress called The Oral History Of Our Time, which he claimed was 11 times as long as the Bible.

Mitchell describes this archive of talk as "a great hodgepodge and kitchen midden of hearsay, a repository of jabber, an omnium-gatherum of bushwa, gab, palaver, hogwash, flapdoodle, and malarkey ..." Gould's ambition is to "put down the informal history of the shirt-sleeved multitude – what they say about their jobs, love affairs, vittles, sprees, scrapes, and sorrows".

This, curiously, notes London-based writer William Fiennes, is one way of talking about Mitchell's Up In The Old Hotel, for which he has written a superb new introduction. "He's my hero!" exclaims the bestselling author of The Snow Geese and The Music Room, who first read Mitchell when he was invited to review the 1998 edition of Joe Gould's Secret, "which came adorned with quotes from Salman Rushdie, Doris Lessing, Beryl Bainbridge, Ian McEwan. I'd never heard of him but I'd always been obsessed with the New Yorker so I'd actually been absorbing him through all the profile writers who had been very heavily influenced by him, such as John McPhee."

Immediately, Fiennes tracked down the rest of Mitchell's work, becoming obsessed. "I was working on The Snow Geese and he really inspired me. When Mitchell introduces you to a character, he gets off the stage. Sometimes he puts himself into his stories but without ego. In Joe Gould's Secret, he's an important character, but his writing touches universal themes through specific details. He uses hardly any metaphors or similes, only declarative sentences that build up like tiles in a mosaic," says Fiennes, adding that that is not to say Mitchell is not a sophisticated writer.

"It's not look-at-me writing and I think that is part of his humanity and his generosity. There's such a love of the world and a Chekhovian streak of melancholy and 'graveyard humour', as well as a love of the variousness of people and the variousness of human experience. He was an amazing reporter, empathising with people's burdens and sorrows yet the least sentimental of writers. He found the gold and the sparkle in ordinary people, because he was a wonderful listener. He understood – like Shakespeare or, again, Chekhov – that there is no such thing as a boring human being."

There is an artistry in Mitchell's seemingly artless work, continues Fiennes, from the selection of details to the scansion of paragraphs. If Harold Ross, creator and editor of the New Yorker and his writers, in the 1930s, conceived and developed the Profile – now a journalistic commonplace – showing that it was possible "to write history about living people", then Mitchell became its greatest exponent. "Of course, he had the gift of time and space – 10,000 words," concedes Fiennes.

Mitchell returned to Joe Gould and his Oral History after Gould's death, following a spell in psychiatric wards, in 1957. He comes to suspect that the book does not exist, that it is a grand figment. As Fiennes points out, Mitchell recognises something of himself in the wanderer who loves the voices of New York City, whose masterpiece remained a marvellous idea. It reminds him of the novel he had planned to write in his twenties, about a young man who, like Mitchell, comes to New York from the South to work as a reporter.

Like one of those mysterious black clams he writes about in Old Mr Flood, Mitchell clammed up – an analogy Mitchell would never have used – after the publication of Joe Gould's Secret. Until his death at the age of 87 on May 24, 1996, he maintained "a profound and elegant silence". Although he went into work at the New Yorker almost every day for 31 years and six months, he submitted no further writing.

"I don't know why more has not been written about the silence," says Fiennes, who harbours an ambition to write a biography of Mitchell. "I don't know whether he suffered from writer's block or was depressed, but the second Joe Gould story is dark. It's almost as if in Joe Gould he'd written a character who came off the page and started stalking him and haunting him. Whatever the explanation, he left a formidable legacy, a substantial body of work that has been incredibly influential."

One of the many writers Mitchell influenced is New Yorker staff writer and author of eight books Adam Gopnik, for whom Joe Gould's Secret is "one of the magic books of American literature". Gopnik met Mitchell in 1986 when he joined the magazine as a young writer. "He was such a benevolent presence. He worked every day, I could hear his typewriter – no one knew what he was working on. We lunched together, although I'd never claim we were friends, he was wonderful company. We talked about Russian literature – he adored Gogol and admired Dostoevsky. As for the silence, over lunch he often said, 'I guess I am getting to be like Joe Gould.' The older I get, the more I find his silence entirely understandable. He'd already written wonderful things – I think he knew how good Joe Gould's Secret is."

Meanwhile, Mitchell is influencing a new generation of writers. The sharp-eyed North Carolina-based essayist John Jeremiah Sullivan, author of Pulphead: Dispatches From The Other Side Of America, emails that he believes Mitchell is "living proof that you could create literature in the highest sense through selection of fact and intensity of observation. When I close my eyes and envision him, I see a giant eyeball with a fedora sitting atop it. He wrote the greatest beginnings of any writer I know. All writers should study his beginnings, the perfect transparency of them – they're open doors that almost any reader can walk through, but they close behind you like traps. Every time I open one of his books, I feel instructed; I find myself learning."

Gopnik once asked Mitchell if there was any common element of style among all the writers who had made the New Yorker so distinctive. "'Well, none of 'em could spell,' he said in his soft Carolina accent. Then he added, 'And none of us, including Ross, really knew anything about grammar. But each one of them – each one had a wild exactitude of his own.'

"A wild exactitude! Isn't that something? It's become my talisman, my motto, a slogan to embrace as a writer."

Up In The Old Hotel by Joseph Mitchell is published by Vintage Classics, £11.99. John Jeremiah Sullivan discusses Pulphead (published by Vintage, £9.99) at the Edinburgh International Book Festival on August 18, www.edbookfest.co.uk, 0845 373 5888