The United States, which hosts the UN, said Iran's candidate, Hamid Abutalebi, was unacceptable given his role in a 444-day crisis in which radical Iranian students stormed the US embassy in Tehran and took 52 Americans hostage.
President Barack Obama had come under strong domestic pressure not to allow Abutalebi into the US to take up his position at the UN in New York, raising concerns that the dispute would disrupt delicate negotiations between Tehran and six world powers, including Washington, over Iran's nuclear programme.
"We have no replacement for Mr Abutalebi and we will pursue the matter via legal mechanisms envisioned at the United Nations," Abbas Araghchi, a senior Iranian foreign ministry official said, according to the country's official IRNA news agency.
"Based on an agreement with the United Nations, America is bound to act according to its international commitments."
The United Nations said it had no comment to make about the US decision.
American law allows the country's government to bar UN diplomats who are considered national security threats. But Obama's potentially precedent-setting step could open the US to criticism that it is using its position as host nation to the UN to improperly exert political influence.
Abutalebi has insisted he served solely as a periodic translator for the Islamist students who seized the US embassy hostages, and that he has since evolved into a moderate figure favouring, like president Hassan Rouhani, a thaw in Iran's ties with the West.
Since an uproar among former American hostages and US politicians over Abutalebi broke out, Tehran has steadfastly stuck by its choice, describing him as a seasoned diplomat who has served in various capacities in Western countries.
However, many Americans retain bitter feelings about Iran over the hostage crisis and many members of Congress - even including some of Obama's fellow Democrats - are deeply sceptical about Tehran's intentions even under the pragmatist Rouhani.