Ivanovic headed in a Frank Lampard free-kick unopposed after 73 minutes as debate raged over whether the Chelsea right-back should have been on the pitch.
Kevin Friend had booked the Serbian defender moments earlier for catching Christian Benteke with a flailing arm in an aerial challenge which infuriated the Villa manager Paul Lambert.
All the attention was on Jose Mourinho's return in Sunday's defeat of Hull City, but the Portuguese wanted to shift the focus to his team, who endured an altogether more difficult test than on Villa's most recent visit to Stamford Bridge.
Villa were beaten 8-0 last December, but were in buoyant mood following their 3-1 opening day win at Arsenal and determined to claim the scalp of another title contender.
Antonio Luna scored late on at the Emirates Stadium and was on target again, only this time into his own net following a Brad Guzan save from an Eden Hazard shot.
Benteke, whose double accounted for Arsenal, scored his third goal of the season with the final kick of the first half to equalise.