Dr Peter Fowlie, Scotland officer for the Royal College of Paediatrics and Child Health, says vacancies for paediatric staff are so severe it would potentially be better for children to travel further, than rely on an unstable service nearer home.
He described locums flying in from Europe for days at a time to provide medical cover and said there were stories raising questions about the suitability of some of them.
Dr Fowlie said: "There just simply are not enough staff, with all the working-hours rules and regulations and the feminisation of the workforce and maternity leave, to support the rotas and out-of-hours cover in all of the paediatric units."
His comments come after gaps in the core paediatric workforce around the country were revealed by The Herald.
Some health boards are operating with half the number of middle-grade children's specialists they need, unfilled posts have been repeatedly advertised, and in the Borders two jobs have been unfilled for two years.
Managers in some areas rely on locums and on occasions consultants working successive days and nights to keep services running.
Dr Fowlie, who works as a consultant in Scotland, said the situation meant there was a risk a parent would one day arrive at their local hospital to find there was no paediatric doctor there.
He said: "It is very possible that there might not be a doctor there or if there is a doctor there it might be someone who does not have the training because if you rely on locums, which many of these hospitals have had to do over the last few years, you cannot be as comfortable with the standard of that locum compared to someone who is part of the established workforce. These people will literally drive up or fly in for a couple of nights work and disappear again."
He said references were obtained for such temporary staff, but added: "There is anecdotal evidence that services have ended up with a locum who with hindsight - has been wholly unsuitable."
The Royal College has set out 10 minimum standards that all paediatric units should meet. Asked how many units would face re-organisation if these were applied in Scotland, he said: "There would probably be discussion around three or four."
The number of consultant paediatricians in Scotland has hardly changed in the last four years and there has been a 20% fall in consultants in community paediatric services.
Health Secretary Alex Neil told the Scottish Parliament last week there was a 34% rise in paediatricians since 2006 but it emerged the number peaked in 2009 and has been around 220 since.
Jackie Baillie, Labour's health spokeswoman, said: "The SNP promised that no services would be cut. To find out that there has been a reduction in paediatric doctors since 2009 really does put at risk the ability to keep local paediatric services open.
"Once again, there is an expert voice raising concerns about the NHS under the SNP's watch."
A Scottish Government spokesperson said: "It is the responsibility of NHS boards to plan and deliver clinical services, including paediatrics, and NHS boards are required to consider relevant local issues and demographic changes and assess the resultant demands and implications for service delivery.
"The use of locum medical staff supports NHS boards to ensure service continuity during times of planned and unplanned staffing gaps."