Willem Alberts, Bismarck du Plessis and Francois Hougaard claimed early tries for the Springboks as England were overrun at Coca-Cola Park.
Youngs crossed twice as England threatened to pull off an unlikely victory after trailing 28-10, but JP Pietersen ended the comeback.
Leicester scrum-half Youngs said: "The first 20 minutes killed us unfortunately and we were always playing catch-up. The boys showed great spirit but at the end it didn't count for much. Now we've lost the series but we've got one more big effort next week."
Youngs' club-mate Toby Flood, who also bagged a try – along with all three conversions and two penalties, said: "In the first half, every time we conceded we kicked off and then we'd be back in our own 22 via a penalty or a good kick that goes a few yards further at altitude. We need to control that better and we can't make those errors."
England head coach Stuart Lancaster added: "The boys are frustrated but we've got to bottle that frustration and take it into next week. We were disappointed with the start, giving them 12 points before we got going. We know we have to be better in all the little details that make the difference in winning at international level.
"We were hanging on by our fingernails in the first half at times. But we grew, the scrum got better and we started to win the collisions and move the ball around."
An England error at the scrum allowed Alberts to claim the opening try. Du Plessis added a second which was converted by Morne Steyn. Flood and Stein traded penalties before Hougaard touched down, with Steyn again converting to make the score 22-3.
England made the most of a rare opening midway through the first half when Youngs' quick tap penalty released Chris Ashton, who sent Flood clear to burst over the line.
Flood converted his own try but Steyn added a long-range drop goal to make the half-time score 25-10. Steyn kicked another penalty after the break but England refused to lie down as Flood converted the first of Youngs' tries.
Steyn slipped over his third penalty before Youngs dived across the line for the second time. Flood added the extras and another penalty but South Africa found another gear when they needed it and Pietersen sealed victory with a try seven minutes from time.