Brown sustained the injury during the team's match against Samoa. He left the field at half-time in Durban and underwent a scan in Nelspruit yesterday.
Scotland team lead physiotherapist Paul McGinley said: "Kelly has a notable ankle injury. He had a scan at a local clinic which indicates ligament damage. He will now be returned to the care of his club medical team at Saracens as he would be unable to play again on the rest of the tour."
However, Glasgow Warriors hooker Fraser Brown is now scheduled to arrive at the Scots' camp in Nelspruit tomorrow to prepare for Saturday's Test against South Africa.
Hooker Brown, who celebrates his 24th birthday next week, is a former Scotland under-20 captain who returned to rugby via Heriot's and the Scotland 7s squad after he had been troubled by injury earlier in his career.
Brown earned a contract with Glasgow Warriors for the coming season after being part of the Warriors' Heineken Cup squad and impressing during appearances off the bench in the RaboDirect PRO12.
Scotland team manager Gavin Scott said: "We wish Kelly a speedy recovery and look forward to welcoming Fraser to the tour tomorrow."
Kelly Brown said: "I'm very disappointed to be heading home but I wish the boys all the best and I'm sure they will do well in the final two games of the tour."
Fraser Brown said: "It's a bit unreal to be honest. A few months ago I was playing in the RBS Premier Division for Heriot's and now I've been called up to the Scotland squad. I'm immensely proud and hugely excited."
Meanwhile, the Scotland medical team continues to treat those players who sustained injuries at the weekend with backs Tom Heathcote and Alex Dunbar "continuing to progress through the return to play concussion protocol."