David Shearer led the centre-left Labour Party for 20 months but appeared to struggle at times in the spotlight. Opinion polls indicate he never gained much support among New Zealanders as preferred prime minister.
Prime Minister John Key continues to enjoy high levels of support after first taking office in 2008.
Mr Shearer will remain in Parliament, but has not yet said if he will contest his seat next year.
He said it was time for a change and added in a statement: "There was no letter, there was no ultimatum, there was no vote.
"But from the soundings I have taken from colleagues, I realise I no longer enjoy the confidence of a number of my caucus colleagues."
Mr Shearer said he came into politics to make a difference and "I believe I have done that".
Labour plans to pick a replacement leader in the coming weeks.