Former Light Blues' Rangers defender Danny Wilson opened the scoring in the 52nd minute with a header from a Billy King corner and the visitors were good value for their lead for most of the second half.
When Gers midfielder Nicky Law levelled near the end of the regulation time it looked like the Govan side would nick a point but the Gorgie side ran up the park and Swedish striker Sow grabbed a dramatic winner with a drive past Cammy Bell.
Gers skipper Lee McCulloch had unfurled the First Division title flag before kick-off but on this display anyone who believes he is sure to be doing something similar in 12 months' time will have to think again.
And on this performance, the Tynecastle side will fancy their chances of going straight up to the Premiership as champions.
Winger David Templeton came in for defender Darren McGregor, the only change to the Rangers side which once again under-performed.
Hearts had former Gers keeper Neil Alexander in goal with other summer signings, defender Alim Ozturk, midfielders Morgaro Gomis and Prince Buaben, and Sow, also in the side.
After the flag celebration the Govan Stand had officially been renamed the Sandy Jardine Stand in memory of the former Scotland player who had served both clubs with distinction, before passing away in April with a long battle with cancer.
After the pre-match formalities it took the players and the crowd a few minutes to get focused on the game.
In the fifth minute a poor pass back from Jambos defender Jordan McGhee had Alexander in all sorts of trouble and he did well prevent Kenny Miller and then Kris Boyd taking advantage.
The home side were making most of the running in a frenetic encounter and should have taken the lead in the 19th minute after good work on the right by Lewis Macleod to play in Miller.
The former Celtic striker's drive was blocked but spun kindly up in the air towards Templeton who had the whole goal to aim for only to somehow send his diving header over the bar.
After near misses at either end from Wilson and Rangers stopper Marius Zaliukas, the Tynecastle side finished the half strongly.
In the 38th minute, after the Light Blues had failed to properly clear a corner, Ozturk could not get a proper connection on a low cross from Wilson.
The ball spun past the far post to the relief of Gers keeper Bell, who moments later had to save a close-range header from Hearts midfielder Sam Nicholson after Sow had delivered a cross from the left.
Kyle Hutton replaced the injured Ian Black three minutes after the interval and while the home side were trying to adjust, they found themselves a goal behind as Wilson climbed inside a packed penalty area to send his header from King's corner past Bell.
With the travelling fans still celebrating, Hearts replaced McGhee with Brad McKay who was immediately booked by Craig Thomson for a foul on Templeton.
In the 66th minute Alexander was carried from the field on a stretcher after colliding with Ozturk while gathering a Law free-kick, to be replaced by another ex-Rangers keeper Scott Gallacher.
Dean Shiels replaced Templeton as Gers boss Ally McCoist tried another route back into the game but there was no onslaught on the Gorgie goal.
However, it looked like Rangers would take a point when Law volleyed in from 12 yards after being set up by Miller but in dramatic fashion Hearts restored their lead through Sow after he had run onto a Nicholson pass, with Hearts seeing out six minutes of added time.