Driving out of Wigtown last Saturday afternoon, I saw a figure standing by the road, thumbing a lift.

How far was he going? Newton Stewart, he said, the bus having left without him. My partner relocated to the back seat, squeezing himself in between enough newly purchased books to start our own emporium, and off we went.

We had been in Newton Stewart already that morning, having been put up by the Wigtown Book Festival in a hotel on its outskirts. In previous years we have been in top-notch B&Bs, an idyllic eco-lodge, and once we stayed with a doctor and his wife, whose delicious cooking converted me from three decades of vegetarianism.

After two days spent at the Wigtown Book Festival, with its indefinable but infectious charm, we were feeling buoyant. Sadly, our passenger was in quite another mood. He was decamping, he said, to get away from the noise of the festival, whose tents and stalls lay under his window.

There is certainly a lot of bustle in the town while the book festival is on, especially on a day like last weekend, when the sun is summer hot. Voices of the authors speaking in the marquees are relayed to the street, so those without tickets can enjoy what is being said for free. A hog roast was being carved for a queue of hungry readers, and people were still talking about the previous night's party, where the disco was taken in hand by teenagers who turned what might otherwise have been a glass-tinkling literary soiree into something more lively.

The last straw for the Wigtown hitch-hiker, it seems, had been the arrival of Joanna Lumley, whose presence turned the high street into a bee hive, photographers, locals and festival staff buzzing as if they had never before seen anyone like her. None of us had, I suppose, but our man was not impressed. Driven to desperate measures, he was heading in search of something for his tea and a good walk home - seven miles - by the coastal path, where he could enjoy some of the most beautiful scenery in Scotland.

Not that he put it like that. Wigtown's locals are not boastful about what their town has to offer, but as this man made clear, if you have got a job, it is a great place to live: a close, friendly community in an extraordinary setting. He, and perhaps others, may not enjoy the hoopla of the 10-day book festival, but almost everyone else we met intimated that they need more, not fewer, such attractions.

Indeed, to judge by the enthusiastic reaction of the book festival's audiences, there is a deep thirst for culture, ideas and entertainment. One woman described Wigtown as invisible to the rest of the country, even the Government which, she felt, directs an unfair amount of money and publicity towards the well-trodden paths of the Highlands and Central Belt. Yet at the same time as a hotelier admitted the town needs more attractions to draw in visitors, he added that they do not want the place over-run.

It is into this delicate situation that the Wigtown Book Festival lands each year. It has always been one of my favourites on the circuit, partly because of the stunning location and the rich history of the area, but mostly because of the atmosphere the festival creates, much of which comes from the audiences, who are thoughtful, straight-talking and humorous. So, too, are the town's booksellers, who are so laid-back that as soon as one steps through the door, one feels stress evaporating.

Falling late in the book festival season, Wigtown is in some ways similar to the Borders Book Festival in Melrose, another bijou event with an exquisite backdrop. Wigtown's remote position near the country's big toe, however, adds an allure few can match. Nor does it take a good hotel for me to want to visit. I would put up in the bunk bed one enterprising bookseller has built in his bookshop, within reach of his higher shelves. I can think of few more inviting places to spend the night.