What they might really mean is that they cannot be bothered, Sir Harry Burns told a conference on sport and exercise in Edinburgh.
"Lots of young people say they don't have time to exercise, lots of people say they don't exercise for health reasons - that seems a bit paradoxical," he said.
"Fewer people say they don't exercise because there aren't facilities there and fewer still say they don't exercise because they're not motivated.
"Do we think they're telling the truth?
"This notion that 'I don't exercise because I don't have time' - is that them not willing to say 'I can't exercise because I can't be bothered?' If you can be bothered, you make time."
People with health problems should understand that "most" conditions can be improved with exercise, he said.
Sir Harry said people are less active for reasons including poverty and reduced levels of active work, calling for an overhaul in the approach to healthy living.
Scotland used to be somewhere in the middle for life expectancy in western Europe but has slipped to the back of the pack in recent decades, he said.
Smoking, diabetes and obesity are bad for the health but fitness is being ignored by the public, he warned.
"They don't understand the significance of physical activity in terms of it being just about the single most important thing you can do for your health," he said.
"Obesity is important as a risk factor but physical fitness clearly goes a long way to creating a healthy population.
"We should encourage people to understand the importance of physical activity for its own sake as well as its benefits for other things."
The focus on health and obesity comes on the day an American academic warned people in Scotland risk "sleepwalking into obesity" if they do not exercise.
Professor Steven Blair, from the Arnold School of Public Health at the University of South Carolina, said a "greater emphasis" had to be placed on physical activity to prevent the obesity problem in Scotland reaching US proportions.
He suggested exercise could be "prescribed" to patients by their doctors or other health professionals.
Prof Blair, whose work is supported with research funding from Coca-Cola, was the keynote speaker at the event organised by the Royal College of Physicians of Edinburgh and the Royal College of Surgeons of Edinburgh.
He claimed physical inactivity had become "the biggest public health challenge of the 21st century", and said people must consider their levels of activity as well as just their diet.
Prof Blair said: "An entire industry has built up around diet, but reducing our dietary intake alone will not solve our problems with obesity. Physical inactivity has become the biggest public health challenge of the 21st century and we have to become more active if we are to stop collectively sleepwalking into obesity.
"In simple terms, we are talking about changing the mind-set from thinking 'I must go on a diet' to 'I must become more active'."
A Scottish Government spokeswoman said: "Increasing physical activity is a key element of the Scottish Government's strategy to tackle obesity. The evidence on the multiple health and wellbeing benefits that can be gained from being more active is overwhelming, which is why we're investing £3 million on physical activity projects in Scotland.
"Our aim is to make Scotland an active nation by encouraging people to make physical activity a part of their everyday lives and we also continue to take decisive action to encourage and enable individuals to make healthier food choices.
"But there is always more that can be done and we'll continue to work with health boards and partners to encourage people to adopt healthy habits and active lifestyles."