Japan Aerospace Exploration Agency (Jaxa) official Yuko Hoshikawa said an automatic countdown for the three-stage Epsilon rocket stopped when an irregularity in the rocket posture was detected.
It is hoped that the rocket will offer a cheaper and more efficient way of putting satellites into space .
Jaxa president Naoki Okumura said the cause of the problem is under investigation and he could not say how soon the launch, on the southern Japan island of Kyushu, could be rescheduled.
He said: "Finding the cause is our first and foremost task. We must examine what happened today and our next launch depends on what we find out."
The Epsilon is the first new rocket design for Japan since the H2A was introduced in 2001.
Space policy minister Ichita Yamamoto said the launch cancellation was unfortunate but that is did not change Japan's policy to set Epsilon as a centrepiece of Japanese space business.
The rocket was due to carry the Sprint-A, the first space telescope specifically designed to observe other planets. It is to be used to watch Venus, Mars and Jupiter.
Epsilon, which uses a solid-fuel propellant, is intended to broaden the range of Japan's space missions.