The man, who cannot be named for legal reasons, is originally from Egypt but moved to Scotland in 2002 after marrying a Scots woman.
They moved to Fife, had three children – a daughter and twin boys – but split up in 2010.
The man sought an order from Dunfermline Sheriff Court to force the children to be brought up as Muslims. However, Sheriff Ian Dunbar ruled no order should be made, given that the children's mother is Christian and they attend a non-denominational school which celebrates Christian festivals.
In a written judgment on the case, Mr Dunbar said: "The day-to-day circumstances of the children should dictate what is in their best interests.
"I have therefore reached the conclusion that it is not in the best interests of the children that a specific issue order be made that they be brought up in the Muslim faith."
The court heard the children's father gave them Muslim names and whispered the call to prayer into their ears when they were born – an act which he says marks them out as Muslims.
The boys were also circumcised in Edinburgh when they were six months old because of their father's religious beliefs.
Their mother has a Christian background but was not a practising Christian until they separated. At that point, she began to take the children to church and Sunday school.
Mr Dunbar said: "I have no doubt at all that if the parties had children in Egypt they would have been raised as Muslims.
"But the children were born in Scotland and are being raised here with their mother as their primary carer. That is a change of circumstance which has some bearing on what may or may not be in the best interests of the children."
He added: "In this case the children are living in a Christian household with a Christian mother.
"They are at the school which has a Christian overlay because of participation in Christian events such as Christmas and Easter.
"If the children were not to be able to participate in these events or if they were to be made to feel uncomfortable participating in these events because of the views of their father, then I do feel that has potential to mark them out as different from the other children and could potentially cause them some distress."
The sheriff ruled the children should continue being raised with Christian values by their mother and be taught about Islam when they visit their father.
Mona Siddiqui OBE, professor of Islamic and Interreligious Studies at Edinburgh University, said: "If the children are going to be brought up by the mother, then they should be allowed to do this in terms of peaceful co-existence.
"The issue here isn't what the court has decided, the issue now is how the parents are going to make sure the children don't feel that they're bereft of one side of their identity.
"The court has made a decision where the court has the right to award custody to any parent, which is very different to an Islamic country where much of the time the right of the custody goes to the father, so I think the real problem is not what the court has decided but the responsibility of both parents now to not let their own hostilities deny the children both sides of two very rich cultures."