What they might really mean is they cannot be bothered, Sir Harry Burns told a conference on sport and exercise in Edinburgh.
"Lots of young people say they do not have time to exercise, lots of people say they do not exercise for health reasons - that seems a bit paradoxical," he said.
"Fewer people say they do not exercise because there are not facilities there and fewer still say they don't exercise because they are not motivated.
"Do we think they are telling the truth? "This notion that 'I don't exercise because I don't have time' … is that them not willing to say, 'I can't exercise because I can't be bothered?' If you can be bothered, you make time."
People with health problems should understand that "most" conditions can be improved with exercise, he said.
Sir Harry, who is stepping down from the role to become professor of Global Health at Strathclyde University, said people were less active for reasons including poverty and reduced levels of active work, calling for an overhaul in the approach to healthy living.
Smoking, diabetes and obesity are bad for the health but fitness is being ignored by the public, he warned.
"They do not understand the significance of physical activity in terms of it being just about the single most important thing you can do for your health," he said.
The focus on health and obesity comes on the day an American academic warned people in Scotland risk "sleepwalking into obesity" if they do not exercise.
Professor Steven Blair, from the Arnold School Of Public Health at South Carolina University, said a "greater emphasis" had to be placed on physical activity to prevent the obesity problem in Scotland reaching US proportions.
Professor Blair was a speaker at the event organised by the Royal College of Physicians of Edinburgh and the Royal College of Surgeons of Edinburgh.
He claimed physical inactivity had become "the biggest public health challenge of the 21st century", and said people must consider their levels of activity as well as just their diet.
A Scottish Government spokeswoman said: "Increasing physical activity is a key element of our strategy to tackle obesity. The evidence on the multiple health and wellbeing benefits that can be gained from being more active is overwhelming, which is why we are investing £3 million on physical activity projects in Scotland."