Herman Nackaerts, deputy director general of the International Atomic Energy Agency (IAEA), said his team had not been granted the access it has long sought to a military site.
A further meeting was scheduled for February 12.
In a separate note sent to IAEA member states about the negotiations, the UN agency said that important differences between the two sides remained.
The absence of an accord, meant to allay international concerns over Tehran's atomic ambitions, will disappoint world powers seeking a broader diplomatic settlement with Iran that would avert the threat of a new Middle East war.
The IAEA's efforts to unblock its stalled investigation into suspected atom bomb research in Iran are separate, but closely linked, to the negotiations between Tehran and six world powers, including the UK, that may resume later this month.
Speaking in Vienna ,Mr Nackaerts said: "We had two days of intensive discussions."
"We could not finalise the structured approach to resolve the outstanding issues regarding possible military dimensions of Iran's nuclear programme."