"Projecting fire control radar is very unusual," Japanese Defence Minister Itsunori Onodera said of the incident, which he said occurred on January 30.
"One mistake, and the situation would become very dangerous."
Prime Minister Shinzo Abe, however, told Mr Onodera it was important to respond calmly and not meet provocation with provocation.
Hopes have been rising for a thaw in ties between Asia's two biggest economies since a chill began in September when Tokyo nationalised a chain of rocky, uninhabited isles in the East China Sea.
In particular, there are hopes for a leaders' summit to help ease the strains that a junior Japanese coalition partner said this week could take place as early as April.
However, deep mistrust, simmering nationalism in both countries and bitter Chinese memories of Japan's wartime aggression mean the road to a summit will be rocky.
The long-running row over the islands, known as the Senkaku in Japan and the Diaoyu in China, has escalated to the point where both sides have scrambled fighter jets while patrol ships shadow each other.
Chinese officials were not available for comment on Japan's complaint but a Chinese spokeswoman urged Japan to stop what she called "provocation".
"We believe that what is most urgent is for Japan to stop provocative actions like regularly sending in ships and aircraft into the waters around the Diaoyu Islands and seek, via talks, an effective way to appropriately resolve this issue," the Foreign Ministry said.