The 29-year-old Team Sky rider tumbled to the tarmac on Tuesday's fourth stage, damaging his wrist.
He started the cobbled fifth stage from Ypres to Arenburg Porte du Hinaut, which took place in torrential rain which slickened the roads.
Froome crashed early on the route before a second crash 85-kilometres in saw him end his defence by taking his place in the back of the Team Sky car.
The stage commemorates 100 years since the start of World War One and features many of the cobbles used in the Paris-Roubaix one-day race 'the Hell of the North'.
Wet weather forced race organisers to remove two of the nine cobbled sections, but Froome's falls came prior even to the first section of cobbles he had been dreading since the route was announced last autumn.
The sad sight of Froome, dominant in winning the 2013 Tour, grimacing by the roadside was reminiscent of Sir Bradley Wiggins' abandonment with a broken collarbone in the first week of the 2011 Tour.
The Tour last took to the cobbles in 2010 in a stage Lance Armstrong predicted would be carnage.
So it proved as Geraint Thomas finished second to Thor Hushovd and Frank Schleck was among those to crash out.
That took place in the dry, but the downpours caused treacherous conditions even prior to the first section of cobbles.
Froome fell on his left side during June's Criterium du Dauphine and again on Tuesday, and then on his right.
A further tumble proved too much for Froome to continue as he exited the Tour.
Team Sky principal Sir Dave Brailsford said prior to the start that Richie Porte would be a protected rider, like Froome, as an insurance policy.
It is likely the Australian will now be Team Sky's leader, although Thomas could also be given a chance to prove his potential.
Froome's fate again highlighted Brailsford's call to omit Wiggins, the 2012 winner, from Team Sky's squad.
Wiggins was ninth in Paris-Roubaix in April and won May's Tour of California but was not selected.
The first cobbled section was due after 87km of racing, with organisers removing sector seven and five - two sections totalling 2.4km - reducing the stage distance to 152.5km, 3km less than originally planned.
Froome joins Mark Cavendish in exiting the Tour after the Manxman's crash on stage one to Harrogate.
Cavendish had shoulder surgery on Wednesday, which rules him out of the Commonwealth Games.
Thomas and Simon Yates (Orica-GreenEdge) were the remaining Britons.
Froome was not the only rider to be unseated on a difficult day, with Marcel Kittel (Giant-Shimano), the winner of three of the opening four stages, also tumbling.
The Kenya-born Briton had been dragged back to the main pack after 42km as seven riders, including world time-trial champion Tony Martin (Omega Pharma-QuickStep) was up ahead.
The frenetic pace and difficult conditions set-up a compelling day's racing, with Froome a victim of misfortune.