A survey of 1500 consumers found four out of five would be concerned if big branches were moved.
The research, by watchdog Consumer Futures, follows controversial plans to franchise 70 so-called Crown post offices, the larger sites usually found on high streets.
The Communication Workers Union has held a series of strikes against the move, warning that jobs and services will be hit.
More than nine out of 10 of those who took part in the latest poll said a town centre location was important.
Almost half of pensioners said it would be difficult for them to visit an out-of-town post office, with one in five believing they would have to rely on a friend or relative to give them a lift.
Consumer Futures urged the Post Office to retain main branches in town centres rather than move them to sites such as hypermarkets.
Spokesman Andy Burrows said: "Moving post offices out of town centres would mean more journeys for people who would normally go to the post office in their lunch hour. Elderly and disabled people would face a problem in getting to a Post Office to collect their pension or benefits. Small businesses and frequent users would have to make additional journeys in order to use the post office."