The East Renfrewshire MP, who moves to the international development brief, was the surprise casualty in a series of moves widely viewed as a purge of Blairites.
Among those also demoted were Liam Byrne, who lost his job at welfare and Stephen Twigg, who was replaced at education.
Labour sources said Mr Miliband wanted to promote talented women and rising young stars to his frontbench team. But the reshuffle was regarded by many as a move to the left.
The Conservatives seized on the changes, pointing to a call last year by the leader of the Unite union Len McCluskey for a number of the key players, including Mr Murphy and Mr Byrne, to be fired.
The Tories said the shake-up meant that, as well as Ed Miliband, Mr McCluskey now had his dream team in the shadow cabinet.
Senior party figures also suggested the move could allow the Labour leadership to change its position on the UK's nuclear deterrent. Mr Murphy has argued staunchly against a scaled-back replacement for Trident, of the kind proposed by the LibDems.
Sources close to Ed Miliband insisted Labour's position on the ageing nuclear weapons system was unchanged.
Mr Murphy said that, as well as taking on the international development brief, he had also been in contact with former Labour Chancellor Alistair Darling and hoped to play a bigger role in the Better Together campaign against independence.
Vernon Coaker, who is considered to have done a good job in Northern Ireland, takes over at defence.
The Labour leader also used the reshuffle to promote a number of women to key positions. They include Rachel Reeves, famously dubbed "boring, snoring" by the editor of Newsnight last month, who takes over from Mr Byrne at work and pensions.
Mr Byrne is now responsible for higher education but has no seat at Labour's top table.
Elsewhere, Maria Eagle has been moved from shadow transport, where she was a keen supporter of the HS2 high-speed rail project, to environment.
The move comes just weeks after Ed Balls, the shadow chancellor, used his speech at Labour's conference to openly question whether the scheme was the best use of £50 billion of public money.
Michael Dugher, a former spokesman for Gordon Brown, was promoted to shadow Cabinet Office minister. Mr Brown's former adviser Spencer Livermore will also return to manage the party's general election campaign.
Overseeing his role will be Douglas Alexander, who remains as shadow foreign secretary, and Mr Dugher.
Labour sources insisted Mr McCluskey had not been consulted about the reshuffle and the changes did not represent a cull of Blairites.
One source said: "I think you will find that we have gone a lot past the days of factionalism. Jim is still a member of the shadow cabinet. Liam and Stephen have got very important roles which they are very happy with."
He also pointed to the appointment of Lord Falconer, seen as a key Blairite, who will advise Mr Miliband on planning for government.
Other women promoted include Emma Reynolds, who becomes shadow housing minister and shadow equalities minister Gloria de Piero. Both will attend shadow cabinet meetings, as will Liz Kendall in her role as shadow minister for care and older people.
TV historian and Labour MP Tristram Hunt has been given the education brief.
Labour sources said almost half - 44% - of the shadow team were now women.
Conservative Party chairman Grant Shapps said: "Len McCluskey is the real winner of this reshuffle."