Colonel Abdulrahman Mohammed al Shami was leaving his home in the capital Sanaa when one of the two unidentified gunmen opened fire on him and fled, a local security official said. The colonel died immediately.
It was not immediately clear who was behind the attack, but authorities have blamed Islamist militants linked to al Qaeda for a string of similar shootings that have killed more than 80 officers in various parts of Yemen over the past two years.
Tackling lawlessness in Yemen, which lies near important oil shipment routes and borders the world's biggest oil exporter Saudi Arabia, is an international priority for the US and other Western countries. It is home to an al Qaeda wing that has planned international bomb attacks.
The Yemeni government also faces a southern separatist movement.