Former Celt Sheridan set the visitors on the way to their historic victory two minutes before the break when he capitalised on slack play by Efe Ambrose to race clear and round keeper Fraser Forster before scoring into an empty goal.
Midfielder Kelly slammed in the second from the spot just after the hour mark when Hoops left-back Emilio Izaguirre was adjudged to have fouled Rory McKenzie and the spirited Rugby Park men held on for their first success in the east end of Glasgow since 1955.
Neil Lennon's men had put in a disciplined, battling performance against Barcelona in the Nou Camp in midweek, only to lose the Champions League clash 2-1 in the final seconds of injury-time.
However, this Clydesdale Bank Premier League performance was at the other end of the scale and they were made to pay for their general lethargy which is sure to infuriate their boss, who had lavished praise on his players for their display against Barca.
Not that this was the exact same side which had come close to getting a point in the Nou Camp.
Into the Celtic team that started in Barca came Adam Matthews, James Forrest, Beram Kayal, Kris Commons and Miku with Georgios Samaras, Scott Brown, Mikael Lustig, Gary Hooper and Victor Wanyama dropping out, the latter three on the bench.
Killie had new signing Mahamadou Sissoko, who has joined the club for a third time following his departure from Udinese, among their substitutes with Sheridan leading the attack and 16-year-old Mark O'Hara in midfield.
After the excitement of Europe in midweek, there was a flatness about the early stages which saw a confident Killie side enjoy a fair share of possession, albeit among some dreadful passing from both teams.
The first real moment of excitement came in the 16th minute when Rugby Park defender Rory McKeown did well to head an Izaguirre cross away for a corner under pressure from Miku but the set-piece came to nothing.
In the 26th minute Joe Ledley missed a sitter by stabbing awkwardly at the ball and watching it fly past the post after Miku had set him up with a header from a Commons cross.
In the 39th minute Charlie Mulgrew split open the Killie defence with a perfect pass for Forrest but the Scotland winger drilled his cut-back across the goal too quickly and the ball sped away from danger.
The Hoops fans grew increasingly disgruntled and they let their feelings be known when Sheridan gave Killie the lead.
Ambrose, who had swapped places with Mulgrew in central defence, was too casual after taking a pass from Matthews and the Irishman got the break of the ball which left him in a race with Forster.
The Hoops keeper came, then stopped, allowing Sheridan to take possession and he was coolness personified as he rounded Forster to walk the ball towards the empty goal before knocking it in.
Parkhead boss Lennon had seen enough from Kayal and he was replaced by Hooper at the interval.
The former Scunthorpe striker should have levelled in the 53rd minute when Forrest set him up with a cross at the near post but he missed the target.
Forster came to Celtic's rescue moments later, tipping over the bar after James Dayton had run through the middle of the parting Parkhead defence before thundering in a drive from outside the box.
The tension was building and Killie's penalty came in the 61st minute when McKenzie had his heel clipped by Izaguirre just inside the box as he set himself to have a shot after driving at the nervous Hoops defence.
Forster was booked by referee Crawford Allan for delaying the spot-kick being taken but that did not affect Kelly who hammered his shot past the big Geordie.
In the 76th minute Izaguirre had the ball in the Killie net after taking a Paddy McCourt pass but was ruled offside.
Moments later, Forrest's cross from the right broke to Tony Watt 10 yards out but with Bell out of his goal and the Rugby Park defence all over the place, he slashed at the ball which flew over the bar.
With four minutes remaining Bell parried Watt's angled-shot from 16 yards but by that time much of the belief had been sapped from the Hoops players and certainly their fans, who slipped off allowing the small band of Killie supporters their moment of glory at the end.
Contextual targeting label:
