The comments, from the architect of Labour's 1997 landslide victory, intensify pressure on the Labour leader over the issue, which was triggered by allegations of vote rigging in Falkirk.
Labour is due to hold a special conference in March to ratify reforms to the party's relationship with the unions.
Mr Miliband wants union members to have to "opt in" in future to allow part of their subscriptions to go to Labour.
But there is speculation the party will scale back the planned reforms to appease the Unite union, its biggest donor.
There is great unease within the party over the changes.
Labour insiders say the party stands to lose up to £9 million a year under the proposals.
The planned reforms follow allegations that Unite "packed" the local Falkirk Labour Party by signing up members without their knowledge.
But Labour was later forced into an embarrassing climbdown over the accusations - after key witnesses changed their statements.
Earlier this month, Unite announced that it could not back a number of the key proposed changes to its relationship with Labour.
Yesterday, Lord Mandelson heaped pressure on Mr Miliband over the issue, linking it to Labour's fortunes at the next election.
Lord Mandelson said: "He's effectively got to win the fight that he started, quite radically, to reform the relationship."
He also rejected suggestions that the changes should be restricted to new joiners or be delayed until after the next General Election in 2015.
The Labour leader did not have an "option", he warned.
"What Ed has got to do is follow the logic of his own analysis, his own diagnosis.
"He said - and I agree with him - that the relationship needs to be between Labour and the individual Labour-supporting members of trade unions, rather than with the general secretaries and their block votes."
Asked if Mr Miliband's electoral prospects depended upon such reforms, he replied: "I think that what many in the public remember is that the leadership was won by Ed on the basis of the trade unions' vote. He's got to sort of distance himself from that."
Lord Mandelson's comments also contained a sting in the tail for Ed Balls, Labour's Shadow Chancellor, who has faced increasing speculation about his future in recent weeks.
The Labour peer said that he "sometimes" thought that Mr Balls was "perhaps better in government than in opposition".
He added, however, that to his mind "that's not a bad thing".
Lord Mandelson also warned that Labour faced a "very difficult minefield" when the findings of the Chilcot inquiry into the Iraq war were published.
And he said that the economy was not "quite as gloomy" as Labour sometimes claimed.
The Conservatives seized on his comments last night.
The Conservative Party Chairman, Grant Shapps, said: "Ed Miliband only got his job because of trade union bosses, and now he is too weak to stand up to them.
"That means Labour would spend more, borrow more, and hit hardworking people with higher taxes - exactly what got us into a mess in the first place.
"Unless Ed Miliband stands up to his union bosses and changes the rules... it will be clear he is too weak to stand up for hardworking people."