The 29-year-old's Omega Pharma-QuickStep team confirmed Cavendish would not start Sunday's second stage from York to Sheffield.
The 25-times Tour stage winner suffered a separated AC joint after colliding with Australian Simon Gerrans (Orica-GreenEdge) as Giant-Shimano's Marcel Kittel won the stage and with it the yellow jersey.
The pain of his injuries means he is no longer able to continue. Cavendish accepted responsibility for the crash, which came as he was pursuing a first yellow jersey of his distinguished career in his mother's home town.
"I'm gutted about the crash today," Cavendish said on Saturday night.
"It was my fault. I'll personally apologise to Simon Gerrans as soon as I get the chance. In reality, I tried to find a gap that wasn't really there.
"I wanted to win today, I felt really strong and was in a great position to contest the sprint thanks to the unbelievable efforts of my team.
"Sorry to all the fans that came out to support - it was truly incredible."
It is the first time since 2008 - when he left prematurely to prepare for the Beijing Olympics - that Cavendish will not complete the Tour. The 2011 points classification winner and world champion won in Paris for four consecutive years from 2009 to 2012.
His exit also deprives the Manxman of an opportunity to win Monday's third stage from Cambridge to The Mall in London, which goes through Essex, where he has a home.
Cavendish's withdrawal means just three Britons remain in the race - defending champion Chris Froome and Geraint Thomas (Team Sky), and Simon Yates (Orica-GreenEdge).