Colin Rice was in JD Wetherspoon pub Campderdown Place underneath Queen Street railway station when he became embroiled in an argument with staff members after draping the flag - which features the red hand of Ulster symbol - over his table.
The 47-year-old, from Belfast, and his friends eventually had their drinks taken from them and were asked to leave, as other Commonwealth fans - who were also displaying flags of their country - looked on shocked.
Mr Rice said: "I'm here for the Commonwealth Games and I've taken in a lot of sports and everywhere I've went I've taken the flag of my country with me.
"There were lots of people with different flags, Scottish, English, South African, and people wearing different tops to match too.
"I put my flag on the table and within about two minutes a member of staff came over and asked me to put it away.
"I asked why and she began linking it to the Orange Order. I explained that I had never had anything to do with the Orange Order and it was the flag of my country, but she still insisted that I remove it."
Mr Rice claimed a further two staff members then attended and again asked him to remove it.
When he tried to explain that he had been at the Rugby Sevens at Ibrox, one worker said 'that's why' - linking the flag to Rangers.
The sports fan added: "The tone was just so offensive. I explained that I'm not even a Rangers fan, but then they snatched our drinks away and we were asked to leave.
"I was outraged. I felt like I might as well have had a swastika on my flag.
"I'm not naive, I know things like this can be a problem at football matches, but this is the Commonwealth Games and this is the flag of a Commonwealth country and these bar staff were too ignorant to realise it."
Mr Rice says he has since had an apology from the bar's manager and been offered a free meal, but he has refused to take it.
A Wetherspoon spokesman said: " We are fully aware of the situation involving Mr Rice.
"A member of staff asked him to put his flag away because it felt that it might offend some people in the pub.
"We totally understand that this flag is the one representing Northern Ireland and we understand Mr Rice's frustration at being asked to put it away.
"A member of staff explained the reason to Mr Rice and he became aggressive towards her.
"The area manager contacted Mr Rice to offer a complimentary meal and to apologise for the incident, which was accepted.
"However, the next day he came in to say that he wasn't satisfied with that.
"At this stage, we don't believe we can do anything else to rectify the situation."