The man, who cannot be named, was imprisoned for two years as part of a four-year extended sentence and was placed on the sex offenders register after pleading guilty in 2010 to a charge of lewd, indecent and libidinous behaviour against his partner's five-year-old nephew.
The man arrived in the UK in 2002 but by the following year his application to remain in the country had been refused, with all avenues to appeal the decision exhausted.
Despite having no legal recourse to stay, he remained in Scotland and struck up a relationship with a UK citizen in 2007, with the couple having a son later that year.
The man is now estranged from his partner and he has not seen his son since November 2009, when he was arrested.
Since his release from prison, he has sought to fend off a decision to deport him from the UK, claiming his human rights and entitlement to a family life would be breached by such a move.
Court papers said he was still prohibited from seeing his son without the approval of his supervising officer but that he had made no attempt to have contact with the child.
He earlier conceded it was not, at that time, in his son's interest to have contact with him.
The man was assessed before his release from prison as "having no victim empathy and to be lacking insight into his offending behaviour".
Part of the man's argument in his latest failed appeal was that it was "normally appropriate and in the best interests of a child that the child should have contact with his natural parents".
In his delivery of the court udgment, Lord Menzies said cases cited by the man did not involve people who had been convicted of offences involving sexual abuse of children.
He said: "As has been stressed on repeated occasions ... the consideration of a child's best interests must be fact-sensitive and involves a detailed analysis of the particular facts and circumstances of the particular case."
The Inner House of the Court of Session - the highest civil court in Scotland - has now rejected the man's fourth attempt to overturn the decision to deport him following conviction.
Lord Menzies said: "I do not consider that the applicant has shown that he has a real prospect of success if this matter were allowed to proceed.
"Accordingly I refuse this application."