They lost two late goals to hosts England after John Souttar had headed Ricky Sbragia's side into the lead.
The young Scots had been aware a win would have left them needing only to beat group whipping-boys Montenegro to qualify and seemed on course to deliver a stunning success after Dundee United defender Souttar scored after the break. They would relent under intense pressure, however, and lost two heartbreaking goals in a six-minute spell.
The Scots were trounced by the Auld Enemy at the same stage of the competition last year and Sbragia can certainly take heart from a much-improved showing 12 months on. His side will bow out against Montenegro on Thursday knowing that they cannot qualify, despite giving the hosts a scare last night and also having drawn against Ukraine on Saturday.
Scotland were given little time to dwell on that earlier performance in Burton and were forced on to the back foot quickly last night. Indeed it took a superb block from Souttar to thwart Norwich City striker Josh Murphy early on.
The young Scots weathered the early storm, though, and a free-kick from Liam Henderson was deflected wide as the visitors started to gain a foothold.
England were almost stunned as they fronted up to the Scots' first real attack. That came five minutes before half time when Ryan Gauld delivered a shrewd pass through the defence for Denny Johnstone to run on to, but the Celtic striker was thwarted by goalkeeper Christian Walton.
Gauld then showed a moment of brilliance when he nutmegged Southampton defender Matt Targett in the penalty area before cutting the ball back to Henderson, but his shot was blocked again.
The half-time whistle brought the match to a halt but the Scots were not interrupted - taking the lead just six minutes after the break. A corner broke for Gauld, who clipped the ball towards the back post for Souttar to nod in.
England responded with vigour and Liverpool youngster Jordan Ibe hit the post with one shot, although the ball took a touch off Reading's Dominic Hyam on its way.
The home side were very nearly made to pay for that moment of profligacy when Scotland substitute Harry Cardwell sent a header over the crossbar with his first touch.
It was a sign of hope, but one which would prove to be fleeting as Chelsea midfielder Lewis Baker smashed the ball into the net after England had again hit the woodwork from a corner. The hosts won the match with seven minutes left as Baker drilled the ball into the net once more.