No organisation has said it carried out the attack. But the country has been grappling with a security threat by al Qaeda-linked militants, who have repeatedly attacked Government officials and installations over the past two years.
Security in Yemen is an international concern. The US-allied country shares a long border with Saudi Arabia, the world's top oil exporter, and is home to one of the most active branches of al Qaeda that has plotted attacks against Western targets.
Security forces retook the compound after killing most of the attackers, the Defence Ministry said in a statement on its website.
A medical source and a ministry official said the gunmen had pulled at least two foreign employees, a Western doctor and a Filipina nurse, into the hospital courtyard and shot them dead in front of local staff.
The attack began as ministry employees were arriving for work when a vehicle exploded at the compound's gate, two sources inside the ministry said. The ministry's official statement made no reference to a suicide attacker.
"The attack took place shortly after working hours started at the ministry, when a suicide bomber drove a car into the gate," the Defence Ministry source said.
The massive blast shook the bustling Bab al Yemen neighbourhood on the edge of Sanaa's old city, a warren of market stalls and stone tower houses decorated with stained glass windows and ornate plasterwork.
Plumes of smoke billowed over the area, where the country's central bank is also located.
"The explosion was very violent, the whole place shook because of it and plums of smoke rose from the building," an employee who works in a nearby building said.
Ambulance sirens and gunshots were heard after the blast as a second vehicle entered the compound carrying armed men dressed in Yemeni army uniforms and exchanged fire with soldiers.
Yemeni President Abd-Rabbu Mansour Hadi later visited the compound, met senior military officials and ordered an investigation into the attack.
"The attackers have exploited some construction work there to carry out this criminal act ... the situation is under control," the ministry said in its statement.
The construction work was taking place near the compound's hospital facility, the statement said.
The Defence Ministry said most of the gunmen were "wiped out" in the ensuing battle but added that two had escaped into a nearby building where they were pursued by security forces.
A military source put the death toll at more than 20, including medical staff, soldiers and gunmen, and said dozens were also wounded.
The Yemeni Health Ministry appealed to citizens to donate blood to help save the wounded.
Violence is common in Yemen, where an interim government is grappling with southern secessionists, al Qaeda-linked militants and northern Houthi rebels, as well as severe economic problems inherited from former president Ali Abdullah Saleh who was forced out of office in 2011.
Islamist insurgents were emboldened by a decline in government control over the country during protests that eventually ousted Saleh.