Zimbabwe-born Moses Mazomba said he was "feeling quite depressed" and had not been eating since being taken to Colnbrook on Thursday evening.
Mr Mazomba, 24, who plays for Cartha Queens Park RFC, had been put on board a Harare-bound flight at Heathrow on Thursday night after losing his campaign to stay in Scotland.
According to Glasgow's Unity Centre, which offers support to asylum seekers, he was escorted onto the plane by "four or five" security guards just before take-off.
A posting on the group's Facebook page said: "On the plane, Moses refused to co-operate and after three or four minutes of struggling with his guards, the pilot ordered his removal from the plane as it was disrupting the flight.
"Moses seems to be fine apart from a sore throat from his shouting."
The group is now fighting to have Mr Mazomba transferred to the Dungavel detention centre in South Lanarkshire so that he can be closer to friends and family in Glasgow.
Unity Centre's Facebook page added: "We all care about him and want to visit him but can't while he is stuck down there."
Mr Mazomba originally came to the UK for medical treatment for an eye condition but when his visa ran out, he was told he had to leave.
He had been living in Glasgow since 2009.
Colnbrook immigration removal centre is described by the UK Border Agency as the "most secure removal centre" within the agency's estate, and was built to category B prison standard.
Speaking from Colnbrook, Mr Mazomba said: "I was badly bruised when they were taking me off the plane on Thursday night, and I have just been in bed for the last few days now." He said he was "feeling quite depressed".
Colnbrook, he said, "is like a prison, but most prisoners complain that it is worse than a prison [in terms of the facilities]. It's terrible. I've been to Dungavel centre and you can't compare it to this one.
"This is really bad, worse than a prison. Built like a prison, but worse."
Asked how long he might be there for, he said: "No one has told me anything.
"When I saw a doctor on Friday morning he said he was going to put in an emergency request for me to be moved back to Glasgow for medical reasons."
Phill Jones, of the Unity Centre, said: "We're trying to get hold of the Colbrook doctor's medical note on Moses and we will be talking to London doctors we work with to see if we can access it."