A third employee at Serco-run Yarl's Wood, in Bedfordshire, was sacked for not taking action when the detainee reported the two men, HM Chief Inspector of Prisons Nick Hardwick said.
It was reported last month police were investigating claims a Roma woman, 23, at Yarl's Wood was subject to inappropriate sexual behaviour from guards.
But inspectors found no evidence a "wider culture of victimisation or systematic abuse" had developed following the new allegations of abuse at the 400-bed centre.
Mr Hardwick said: "We were concerned to find that two staff had engaged in sexual activity with a female detainee, something that can never be less than abusive given the vulnerability of the detained population, and these staff had rightly been dismissed."
Yarl's Wood holds mainly single women but also holds a number of adult families and there is a short-term holding facility for men. Inspectors said more female staff were needed urgently.
John Tolland, Serco's contract director, said: "As the HMCIP report says, we need to make further improvements and we are already working closely with the Home Office to implement their recommendations."