Prosecutors are seeking a homicide conviction and 12-year jail sentence against José Jiménez, who is alleged to have pushed Stephen Mallon 16ft to his death from a terraced balcony in the small Andalusian town of Cómpeta in June 2009.
Mr Mallon, a roofer, originally from Rutherglen, Glasgow, had been celebrating with three friends and his twin sons Carl and Peter ahead of their 17th birthday when a fight broke out with young Spanish local men in La Estrella pub, Cómpeta, where the family had a villa.
José Jiménez told the court he did not join the mass brawl but went to a nearby terrace with his brother, also a defendant.
Mr Jiménez claims Stephen Mallon came up to the balcony and suddenly punched him twice while he defended himself with his arms covering his face.
"I might have given him a small slap," Jiménez admitted to the court. He insisted he hid not push Mr Mallon.
Mr Mallon's sons told the court they feared for their lives in the attack, which left them with injuries. They claim they were attacked for being foreigners.
Scottish and English expats living in the mountain village of Cómpeta, where a third of the 4,000 population are northern Europeans, are adamant there are no racial tensions.
The trial continues.