Wales' Steven Morison (left) and David Edwards stand dejected after Serbia's second goal
Woeful defending played a major part in Wales' heaviest defeat since 2006 and Aleksandar Kolarov and Zoran Tosic had the hosts 2-0 up inside 25 minutes.
Gareth Bale gave Wales hope with a stunning free-kick but Filip Duricic got a third for the hosts before the break and Dusan Tadic, Branislav Ivanovic and Miralen Sulejmani completed the rout.
Wales started positively with Ashley Williams heading over from an Aaron Ramsey corner in the sixth minute and Gareth Bale posing problems down the right flank.
But it took a block from Adam Matthews to keep out a free header from Nastasic at the other end, before Darcy Blake came close to putting through his own net with a sliced clearance.
Wales fell behind after 15 minutes when Ivanovic won a free-kick on the right edge of the area.
Kolarov, who had been critical of Wales in the build-up to the game, stepped up to curl a left-footed effort over the wall and beyond Boaz Myhill.
And the deficit doubled in the 24th minute with Kolarov this time the creator.
The left-back was somehow allowed to pick his way past Williams and Chris Gunter to get away a shot from near the byline. Myhill blocked that effort but Zoran Tosic converted the follow-up.
It would have been worse for Wales had Gunter not blocked Tadic's goalbound effort with the visitors looking vulnerable almost every time the hosts came forward.
Bale gave them a lifeline just after the half-hour mark as he picked himself up after being fouled by Milan Bisevac to thump home a superb swerving free-kick from almost 30 yards.
It was the first goal of manager Chris Coleman's stint in charge, but Wales' defensive frailties resurfaced to undermine their attempt at a recovery.
Blake's awful defensive header allowed Tadic to play in Filip Duricic and he held off Matthews and fired high into Myhill's net.
Coleman withdrew Matthews, who had endured a torrid time up against Tadic and Tosic, and David Edwards, bringing on Sam Ricketts and David Vaughan.
But it made little difference. Aaron Ramsey gave away the ball cheaply in midfield and only a block from Gunter prevented some lovely interplay between Markovic, Tadic and Duricic producing the latter's second of the night.
But Tadic made no mistake in the 55th minute. Ivanovic dispossessed Bale and beat Ricketts down the right before sliding the ball across to give Tadic a simple finish.
Tosic went close to grabbing his second after more good work from Duricic, with Ricketts making a vital intervention.
Wales were once again guilty of making problems for themselves when it came to the fifth goal.
Blake surrendered possession on the edge of their own area, and home skipper Ivanovic capitalised to slide a right-footed effort into the bottom corner.
Substitute Miralen Sulejmani then rounded off the scoring as his deflected strike looped in over Myhill.
The goal made this Wales' worst result since the 7-1 loss to Holland in 1996, on the most miserable of nights for the visitors.