Derek Adams' County were on course to stop a run of three straight defeats after Jake Jervis' 50th minute equaliser had cancelled out Nadir Ciftci's first-half opener at Tannadice.
But the Highland side were left to rue another loss as Erskine capitalised on a mistake from County goalkeeper Antonio Reguero to lash home United's crucial 87th-minute strike and give the Taysiders three wins from their opening four games in the wake of losing 6-1 away to Celtic the previous weekend.
The hosts created the first real chance in the sixth minute. Ciftci's attempted shot inside the box was blocked by Scott Boyd and the rebound fell to United captain John Rankin, whose powerful shot was turned onto the bar by Reguero.
The hosts took a 19th-minute lead after Yoann Arquin fouled Ciftci. Midfielder Paul Paton floated a free-kick deep into the visitors' penalty area and Ciftci, lurking alone at the far post, rose unchallenged to send a header back across Reguero and into the corner of the net for his first goal of the season.
The home team came close to making it 2-0 just a couple of minutes later when Ciftci unleashed a long-range effort which Reguero did well to turn round the post.
With 10 minutes to go before half-time, County attempted to pull themselves back on level terms when on-loan Cardiff man Filip Kiss let fly from outside the penalty area, but saw his shot blocked by United defender Callum Morris.
Soon after, Morris was booked for hauling down Graham Carey on the edge of the box, but Richard Brittain's free-kick came back off United's defensive wall.
Jim Fenlon was then booked for County with half-time approaching, for scything down Gary Mackay-Steven as he was on the verge of heading into the penalty area.
United made a strong start to the second half with Ciftci releasing Mackay-Steven down the left in 48 minutes - but his low effort from a tight angle hit the side-netting.
In the 50th after Ciftci's shot from the edge of the area was blocked and the visitors quickly launched a counter-attack which proved decisive.
Joe Cardle advanced towards the halfway line before passing down the left to Carey who, in turn, crossed for Jake Jervis to calmly side-foot home from eight yards, unchallenged by any United defender.
It was end-to-end stuff now as both sides looked to get in front and Mario Bilate came close for the hosts as his left-foot shot from inside the box went just past Reguero's far post.
However, County missed a sitter on the hour. Cardle passed to Uros Celcer, whose cross was met by Arquin - but he headed wide from a few yards with the goal at his mercy.
In the 77th minute, Jervis had a golden opportunity to give his team the lead - but from close-range he volleyed well over from Carey's cross with only Cierzniak to beat.
And it proved costly when Erskine won the points with three minutes left, lashing the ball home from 10 yards after Reguero could only half-punch Gary Mackay-Steven's corner clear.