David Wolffe's exit comes just days after it emerged Simon Fox, who has been at the helm for six years, will be replaced by Trevor Moore, the former boss of camera retailer Jessops.
The changes have added to jitters about HMV's performance, with the group expected to report full-year losses of £16 million later this week.
Seymour Pierce analyst Kate Calvert said: "This is likely to result in speculation that trading remains under pressure and that the business may not return to profit in the full year."
She said HMV had failed to gain from the demise of Zavvi in 2008 and will continue to come under pressure from online players and supermarkets.
HMV said it hopes to return to profit in the current year as it benefits from selling more video games following the collapse of Game. It secured its immediate survival earlier this year after a refinancing deal with banks and better terms with key suppliers such as Universal Music.
The shares were more than 4% lower yesterday.
Mr Wolffe, who joined HMV from ITV Studios in January 2011, will stay on until a replacement is found.
HMV has seen its share price plummet from 140p before the financial crisis to 3.5p, giving it a market value of about £15m.
As part of its turnaround plans, it sold its Waterstones book chain for £53m in 2011 to Russian billionaire Alexander Mamut and offloaded the Hammersmith Apollo venue.
The company hopes Mr Moore, by building closer relationships with key suppliers, will breathe life back into HMV in a similar way to how he revived Jessops.