Truck driver Peter Nettleton, from Sydney, said he was "gutted" to see a front page photo in The Australian newspaper of his grandson lifting the head by the hair with both hands.
The newspaper said the image was taken in the northern Syrian city of Raqqa and posted on Twitter by the boy's convicted terrorist father Khaled Sharrouf.
Mr Nettleton, father of Sharrouf's Anglo-Australian wife Tara Nettleton, said he thought his five grandchildren had been living in Malaysia while Sharrouf fought with the Islamic State army in Syria and Iraq.
"I'm scared for the children," Mr Nettleton said. "What life are they going to have now? Can't the government do something to pull these kids away from that man?"
Sharrouf, 33, also posted a photograph of his three sons posing with him in matching camouflage fatigues and armed with assault rifles.