Mr Hester takes on the role with immediate effect and will be paid an annual salary of £950,000 with the potential for a long-term incentive scheme shares worth up to £2.9 million this year on top of a potential annual bonus.
He takes over from chairman Martin Scicluna, who has been acting in an executive capacity on an interim basis following the resignation of former boss Simon Lee in the wake of the group's Irish crisis when a £200 million black hole was discovered in the division's finances.
Mr Hester left RBS at the end of September after five years at the helm.
Mr Scicluna, who reverts back to the post of non-executive chairman at RSA, said Mr Hester had an "outstanding track record of transforming the performance of businesses".
He added he will "work quickly to take the group forward".
Mr Hester said : "RSA has grown into a global insurer with market leading businesses, talented and engaged staff and long-standing partners and customers.
"However, the challenges of recent months have demonstrated that we have not lived up to our stakeholders' expectations and performed to our true potential."
The group has been rocked after its Irish woes contributed to three profit warnings in six weeks in the run-up to Christmas, which led to Mr Lee's departure.
RSA recently sought to draw a line under the Irish scandal after an independent investigation found "inappropriate collaboration'' among the division's managers led to the financial black hole.
The report, led by PricewaterhouseCoopers, confirmed the issues were isolated to Ireland and alleged executives sought to deliberately sidestep group governance controls.
RSA said Irish finance boss Rory O'Connor and the division's claims director Peter Burke were fired over the issue last month following a disciplinary and appeals process.
Irish chief executive Philip Smith had already resigned after being suspended.
But RSA revealed yet more troubles in January as it said the severe flooding in the UK and Ireland over Christmas, combined with ice storms in Toronto, would further hit 2013 figures.