The men had been held at the US naval base at Guantanamo Bay in Cuba since 2002, and were classed by the Pentagon as "high-risk" and "likely to pose a threat".
Two are also implicated in the murder of thousands of minority Shi'ite Muslims in Afghanistan, according to the US military.
They were released in a swap for US army sergeant Bowe Bergdahl, the sole American prisoner of war held in Afghanistan, who was flown to a US military hospital yesterday.
"They will definitely go back to fight, if health-wise they are able to go," said a top official at Afghanistan's spy agency who asked to remain anonymous.
"They will be very dangerous people, because they have connections with regional and international terror organisations around the world."
The Taliban denied the prisoners would return to battle but said the swap should not be regarded as a gesture of good will or a step towards the revival of peace talks between Islamist insurgents and the Afghan Government.
"This is purely a negotiation between the Taliban and the Americans... It won't help the peace process in any way, because we don't believe in the peace process," said Taliban spokesman Zabihullah Mujahid.
He added that the prisoners would return to their families and live in Qatar - the Gulf emirate that brokered the exchange - where they would lead normal lives.
The swap comes days after the US announced plans to withdraw all but 9800 troops from Afghanistan by the end of the year and the rest by 2016. Many senior Afghan officials and diplomats say the drawdown will happen much faster than expected and reflects a US desire to disengage from Afghanistan as quickly as possible.
The prisoner swap is further evidence of US efforts to tie up as many loose ends as possible before leaving, diplomats say. "They have made a mess of things," said one Western diplomat, frustrated with the pace of the drawdown.
In a further reflection of the rupture in relations between the two countries, the US did not inform President Hamid Karzai's government about the swap in advance. His palace declined to comment.
On the streets of the capital Kabul many expressed anger at the decision to release the five men, a contrast with scenes of celebration in Sgt Bergdahl's home town in Idaho.
"This decision showed that the region, Afghanistan and its people aren't worth anything to American government," said Gul Mohammad, a high school teacher.
"Otherwise, why would they swap a useless army soldier who broke the law with the five most dangerous Taliban fighters?"
Some among Afghanistan's security forces also expressed unease about the release, which comes as the Taliban's summer offensive gathers pace ahead of a second round of voting in the presidential election on June 14.
"This act will boost the Taliban's morale and encourage them to fight harder to capture foreign soldiers," said army colonel Asadullah Samadi. "Now they are confident that their efforts won't be wasted."