Amido Balde missed a decent chance before Christopher Nothe slotted the ball past Craig Gordon five minutes before the break.
Tony Watt twice came close as Celtic fought back and the lively Paul McMullan won a penalty with three minutes left, but Atajic shot over the bar.
Israeli midfielder Nir Biton captained Celtic while Gordon and Lukasz Zaluska shared goalkeeping duties.
Hamilton needed two late goals and two penalties to come from behind and beat Shildon 3-1.
Jason Scotland levelled from the spot just before the break and Danny Redmond finished well with nine minutes left before Mickael Antoine-Curier converted the second penalty.
Elsewhere in the North East of England, St Mirren needed two injury-time goals to defeat Blyth Spartans 4-2.
James Marwood helped Saints to a 2-0 lead but they were pegged back until Ross Caldwell and Mark Williams scored late goals. Ross County drew 1-1 at Nairn County.