The warning comes after a Sierra Leone mountain biker vanished from the Athletes' Village after two team-mates tested negative for Ebola.
Reports suggest up to 30 team members are considering extending their stay in Glasgow beyond the closing ceremony on Sunday.
The team arrived for the opening ceremony 10 days ago with 26 athletes and other officials. It is thought they have three-month visas, although when these expire is not clear. The Home Office would not comment.
Champion rider Mohamed Tholley, 25, has not been seen since cyclist Moses Sesay and table tennis player Samuel Morris were taken to hospital for the checks.
He did not turn up to compete in Thursday's cycling time trial event. Tholley's room key was later found in the room. Senior Sierra Leone team members believe he may have absconded over fears about the deadly disease.
Winston Crowther, president of Sierra Leone national cycling, said: "My fear is the fact that Mohamed Tholley left his room key behind means he will not be coming back."
Unisa Deen Kargbo, told The Times that the delegation are worried about the situation at home, where a public health emergency has been declared.
Mr Kargbo said: "Many people are thinking whether or not to go home now. Everybody is worried and many of them don't want to go home now because of the Ebola.
"We have held several meetings with them, but they are still worried. This virus is spreading around our country and everyone is at risk of catching it. The problem is, if they want to stay on after the Commonwealth Games end, who will take care of them?
"They will have no accommodation, no work. How do they meet their needs? How will they get themselves employed?"
He said that the athletes will be forced to return to Sierra Leone if these issues are not addressed.
The Sierra Leone team were one of the main topics at today's daily Games briefing.
Asked if there would be a possibility for them to stay on at the athletes' village, Glasgow 2014 communications director Jackie Brock-Doyle said: "The athletes' village closes on Wednesday and is being decommissioned on Thursday, so it will become a building site within a matter of hours.
"We had a meeting with the chef de mission this morning and all his plan to take his entire team back to Sierra Leone on their flight on Tuesday still stands."
She added: "The chef de mission knows where cyclist (Mohamed Tholley) is. The cyclist is not missing.
"He is aware of where the athlete is and the police are not involved. Police Scotland issued a statement last night saying they have not been asked to find him."
Yesterday the head of the World Health Organisation (WHO) warned the outbreak is out of control but can be stopped with more resources and tougher measures. It is the worst since the disease was discovered in the mid-1970s, with 729 deaths in four different countries.
Sierra Leon cyclist Sesay, 32, last week attended Gartnavel Hospital and was tested for Ebola, but released after being quarantined for four days.
Table tennis player Morris, 34, is also understood to have been tested negatively. The Brownlee Centre for Infectious and Communicable Diseases at the hospital has been identified as the receiver facility for any suspected Ebola cases.
Ebola is a severe, often fatal illness, that affects humans as well as primates. The virus is transmitted through close contact with the blood, organs or other bodily fluids.
Once a person becomes infected, the disease can spread through contact with a sufferer's blood, urine, saliva, stools and semen. The Ebola virus is fatal in 90% of cases. There is no vaccine and no known cure.
A spokeswoman for Health Protection Scotland said Sesay's was an isolated incident and no other athletes were tested.
* A 20-year-old woman of unknown nationality has been reported missing from the athletes' village.