Mr Fahim was an ethnic Tajik who was the top deputy of Ahmad Shah Massoud, the charismatic Northern Alliance commander who was killed in an al Qaida suicide bombing two days before the September 11 attacks in 2001.
President Hamid Karzai's office said Mr Fahim, who held the rank of field marshal, died from an illness in Kabul. The exact cause of death was not immediately known. Mr Fahim had survived several assassination attempts, most recently in 2009.
His longtime friend and Afghanistan's ambassador to Spain, Masood Khalili, said Mr Fahim "was not feeling good", adding: "He had diabetes. He had had two heart operations and three times he had gone to Germany for check-ups."
Mr Fahim served as defence minister in Mr Karzai's first administration and most recently was the first of two vice presidents.
But he was best remembered as a former warlord who fought against the Soviets when they occupied the country, and for taking part in the bitter internecine fighting that marked the early 1990s.
In a televised address to the nation, Mr Karzai called Mr Fahim his close friend and brother, adding: "No one can replace him. It is a loss for all of us."