The survey also found only two out of 10 voters are satisfied with the Labour leader's performance.
A total of 34% of people who voted for the Labour Party in 2010 say he should not lead the party into the 2015 general election, while just 46% say he should, the ICM poll carried out for the Daily Mirror found.
While 21% of voters said they were satisfied with the way he is leading the party, 42% said they were dissatisfied. Among Labour voters, 45% said they were satisfied and 31% dissatisfied.
Mr Miliband will find it harder to dismiss criticism of his management style, following publication of the poll by the left-leaning tabloid, which has traditionally backed his party.
His summer leadership crisis has not been helped by criticism from within his own ranks that the Opposition has failed to stop the Conservative Party from dominating the news agenda.
Mr Miliband's approval rating is particularly low among older and male voters. Of those polled, 61% of pensioners said they were unhappy with his leadership, while 48% of men say he was not up to the job, compared to 37% of women.
Given a choice of eight Labour MPs, Mr Miliband came out on top with 16%, when asked who would be the best person to lead the party. Among Labour voters, this figure was 28%, 10% ahead of his brother David Miliband.