Neville, a former player whose career included stints at Manchester United and Everton, was criticised for his lack of emotion and "monotone" style during the game.
Many viewers took to Twitter to criticise him, with several joking that England physio Gary Lewin who was stretchered off after injuring his ankle had actually "fallen into a coma" listening to Neville.
A BBC spokeswoman said there were 445 complaints after Saturday night's game, which pulled in a peak audience of 15.6 million viewers.
Neville told Radio 5 Live: "Co-commentary is harder than I thought it would be...I will get better...I'm glad I helped everybody sleep!"
Fellow broadcaster Danny Baker was among his critics, but said the BBC should share the blame.
He said: "Phil Neville has acknowledged he wasn't great during England commentary. But what were the BBC doing giving him THAT game to 'learn his craft'?"
By contrast, Neville's brother, Gary, is regarded as one of the sharpest analysts in football broadcasting.
The BBC said Phil Neville, who has received broadcasting training, was "an important, well-respected member of our team" and would "continue to play a key role throughout the tournament".
Neville is not the only broadcaster to be on the receiving end of criticism.
His BBC colleague Jonathan Pearce was mocked on Twitter after he appeared to be confused by the use of goal-line technology during France's game with Honduras and then mistakenly said France had scored during another attack.
But a BBC spokeswoman said no complaints had been received about Pearce.