Alan McKinnon has been taken off the roads for a month and instructed to take a customer courtesy course after Glasgow City Council accepted he had breached several conditions of his licence.
The local authority took the decision despite appeals from those who made the complaint against him for no sanctions other than an apology.
The licensing committee also heard there had been use of the term "Hun", a derogatory term for Rangers supporters also considered sectarian by many, by both a passenger and Mr McKinnon.
In an incident which made global headlines, Mr McKinnon, from Bridgeton in Glasgow, was accused of offloading four passengers at the side of the road at 1am on a Monday before Christmas after they objected to his demand that they stop speaking in Irish.
A complaint by the nephew of two of the passengers, brothers Anthony and Joseph Blair from Donegal, was then lodged with the council's taxi enforcement unit and referred to the licensing committee by the authority's head of land and environmental services.
At yesterday's hearing, a statement by the Hampden Cars driver was read. It stated: "I collected three males. During the journey one of the party started to sing/chant: 'There's Only One Glasgow Celtic.' I asked the chap to stop. The chap said: 'You must be a Hun then.'
"I said: 'That's right, I'm a Hun.' Two of the party began speaking in a different language, Gaelic I think. I asked them to stop but they continued."
But witness statements from three of the passengers stated that Mr McKinnon told them that 'well, when you're in Britain it's English that's spoken'.
The driver's lawyer, Euan Robertson, said he had believed the pair were "plotting" and only removed them from the cab as they did not comply with his demand to refrain from speaking Irish.
Mr Robertson was also quizzed on how his client usually handles football supporters and drunkenness in his cab.
Councillor Bill Butler said: "'Hun' is an unacceptable term, that's agreed. Perhaps it was an inappropriate attempt at banter."
Mr Robertson confirmed that nothing else had been said, adding that "Hun" was said "not in an aggressive tone".
Committee chairman Councillor Chris Kelly said: "The conversation was not threatening but there was a plot in some way against him. Also that it was his expectation that they would desist from speaking in Irish. I am concerned about the use of the words 'did not comply'.
"The particular comment was meant in a non-aggressive manner. Does Mr McKinnon have a difficulty dealing with passengers?"
Mr Robertson insisted that his client had no difficulty in dealing with passengers and that after 19 years of service the incident was an "aberration".
In his statement, Joseph Blair said: "I would not want the driver to lose his licence or job as we all make mistakes, but he should learn from them and not be punished."