Watchdogs criticised St Margaret's Adoption and Family Care Society, accusing it of discrimination and breaking equality rules.
It followed a complaint from the National Secular Society, which described the agency's policy as "sectarian decision-making" and "publicly financed crude discrimination".
Sources in the church said the agency depended on charitable donations and the loss of its status could shut the organisation after almost 60 years.
The agency will meet with lawyers later this week to discuss the ramifications before fully responding.
In an announcement yesterday, the Office of the Scottish Charity Regulator (OSCR) said: "The criteria it applies to people who inquire about assessment as prospective adoptive parents discriminate unlawfully against same-sex couples.
"The charity's preferred criteria prioritise couples who have been married for at least two years: marriage is not available to same-sex couples and this constitutes direct discrimination."
Scotland has two Catholic adoption agencies, accounting for almost 20% of all adoptions in the country and following the Church's teaching on marriage by only placing children for joint adoption by married couples.
However, discrimination legislation makes it unlawful to refuse to provide a service on grounds of sexual orientation.
When the controversial laws were passed faith-based adoption agencies warned they could be forced to end their work, with several south of the Border having already done so.
Four years ago First Minister Alex Salmond said he wanted to find a way to save Catholic adoption agencies from being forced to close because of homosexual discrimination laws.
However, the regulator's judgment appears to give St Margaret's little option but to comply with equality legislation or face an end to its charitable status.
The decision states: "OSCR finds the exceptions provided under the Equality Act 2010 for discrimination by charities and religious bodies do not apply to the charity.
"Since it is discriminating in breach of equality law, OSCR finds access to the benefit the charity provides in assessing prospective adoptive parents is unduly restricted. OSCR also finds this discrimination causes disbenefit to same sex couples.
"For these reasons OSCR finds the charity does not provide public benefit, and it therefore fails the charity test."
While St Margaret's is primarily an adoption agency, it also provides support to families after an adoption order has been granted, and often in later life.
Alistair McBay, spokesman in Scotland for the National Secular Society, said the rules "restrict the pool of adoptive parents".
He added: "We hope St Margaret's will continue to fulfil its valuable role, even if it has to sever its connection with the Catholic Church."
A St Margaret's spokesman said: "We have been informed by OSCR of their findings. We will fully examine the contents of their determination and take appropriate legal counsel before responding."