A US defence official said the five were American.
A Nato statement said: "The casualties occurred during a security operation when their unit came into contact with enemy forces. Tragically, there is the possibility that fratricide may have been involved.
The incident is under investigation.
"Our thoughts are with the families of those killed during this difficult time."
If confirmed, it would be one of the most serious cases involving coalition-on-coalition fire during the nearly 14-year Afghan conflict.
One of the worst came in April 2002 when four Canadian soldiers were killed when an American F-16 dropped a bomb on them near a night firing exercise in southern Kandahar.
A senior police official in southern Zabul province said the soldiers in the latest incident might have been killed when they called for close air support during a joint operation by Afghan and foreign forces.