Motherwell, who led 2-0 at Fir Park before being pegged back in injury-time, again got an early goal when Steven Hammell scored a rare header but Keith Lasley gave away his third penalty of the tie and Olafur Finsen levelled in the 37th minute.
Lionel Ainsworth volleyed Motherwell ahead midway through the second half, but Rolf Toft took the game into extra-time with an impressive strike five minutes from time and Atli Johansson scored a spectacular winner in the second period of extra-time to secure a 5-4 aggregate win.
Motherwell forced a succession of early corners on the artificial surface and the pressure paid off in the 12th minute when Hammell powered a header into the top corner from Ainsworth's delivery.
Lasley came close from another corner when he fired just wide and Motherwell were looking comfortable but Hammell had to throw himself in front of Toft's shot.
And Lasley was soon on the end of a controversial decision by Greek referee Antonis Giachos, who ruled he had brought down Johansson in the box, although the award might easily have gone in Motherwell's favour after the Stjarnan player brushed into the midfielder's body and fell over.
Finsen, just as he did twice at Fir Park, converted the penalty.
Lasley was determined to atone and almost did so on the stroke of half-time with a volley that was well-stopped by Ingvar Jonsson.
Motherwell almost fell behind early in the second half but Dan Twardzik came off his line quickly to foil Toft.
But Stuart McCall's side came roaring back and Lasley came close again following another corner when his shot was deflected over.
Craig Reid's cross was then deflected on to the post and Ainsworth's delivery was just turned wide as the hosts continued to live dangerously.
Stjarnan broke and Pablo Punyed shot wide just before Ainsworth put Motherwell ahead for the third time in the tie.
Josh Law, who scored twice in the first 19 minutes at Fir Park, sent in a delightful ball and Ainsworth volleyed home from close range.
Motherwell almost squandered their lead again when Hordur Arnason got in behind Reid but he made a hash of his finish.
McCall, knowing his men would at least get to extra-time unless Stjarnan scored twice, brought on Fraser Kerr and Zaine Francis-Angol for Ainsworth and Law in the 74th minute for fresh legs and more defensive strength.
But there was another twist when Toft levelled with five minutes left as he shot into the top corner from the edge of the box.
Motherwell lost Stuart Carswell to injury early in extra-time as Jack Leitch came on and Jonsson made a great double stop from Lasley and Sutton, but the hosts then missed two good chances.
And Stjarnan, who are unbeaten in the league after 12 matches, won it with seven minutes left when Johansson fired a long-range volley in off the bar to set up a tie with Lech Poznan.