The appointment brings in another executive with substantial Australian experience to head the retail operation at a time when the business is becoming increasingly important to RBS.
Mr Matheson, who has been RBS's acting chief executive for UK retail since September, said: "I look forward to leading the business through this next important phase, to build the best retail bank in the UK."
Mr Matheson will oversee RBS's 300-strong branch network in Scotland. The Edinburgh bank has 2050 branches UK-wide, including 316 which it has been ordered to sell by the European Commission.
It is expected Mr Matheson will continue on the path pursued by Mr McEwan, who developed plans to invest in technology and in more "points of presence" for customers in locations such as shopping centres and railway stations, including automated teller and cash-deposit machines. This is also likely to see around 10% of branches closed by the part-nationalised lender.
RBS reports there has been a 30% drop in branch transactions since 2010 as customers make more use of online banking and the telephone.
Since September, Mr Matheson has overseen the drive to simplify RBS's retail banking operations by removing processes and simplifying its product range.
Robert Webb, associate professor of banking at Nottingham University, said: "He has proved himself to be a safe pair of hands.
"He knows the edicts of lending: treat your customers well, make your offices look nice, push the internet and do not make any silly decisions.
"It is really boring but that is what he will be doing. If RBS make another mistake, that is it over for them."
The appointment comes as RBS accelerates the scaling-back of its investment bank and is preparing to hive off its US arm, leaving the UK retail bank a more significant part of the business.
Mr Matheson, although originally from Scotland, has spent much of his career in Australia. Prior to joining RBS, he was group executive of retail for St George Bank, part of Westpac, following 11 years at Citigroup, where he was ultimately chief executive of Citibank Australia.
Mr McEwan, a New Zealander, used to head Commonwealth Bank of Australia's retail arm. His predecessor Brian Hartzer came from another Australian lender ANZ.
Dr Webb said the Australian experience is of particular interest to UK banks because these "boring" institutions dodged many of the mistakes that hobbled British banks in the credit crunch
"We are in a period now, and have been for the last five or six years, where banking is looking remarkably similar to the time before they started to expand into all the other shadow banking and investment banking activities," he said.
Mr Matheson is not a career banker, having started his career at consumer goods company Procter & Gamble.
He has been at RBS since 2010 and was previously managing director of products and marketing.
He will be based at RBS's Gogarburn headquarters on the edge of Edinburgh.