The bank, which is 80% owned by taxpayers since being bailed out in 2008, is the latest financial institution to state it will not finance expansion of Abbot Point coal terminal near Bowen, Queensland, following similar moves by HSBC and Deutsche Bank.
The terminal will export coal from nine major mines planned for Queensland's Galilee Basin and its expansion involves dumping three million cubic metres of dredged material in the Great Barrier Reef world heritage site.
Environmental campaigners and scientists fear the dumping of dredged material could damage the ecosystem of the reef, with sediment potentially smothering corals and seagrasses and exposing them to pollutants and elevated levels of nutrients.
In a tweet, RBS said: "We are not involved in financing the expansion of Abbot Point & have no plans to be involved in the future."
The move by RBS came as world heritage body Unesco deferred a decision on whether to add the Great Barrier Reef to its World Heritage in Danger list until 2015.
If listed, it would be one of only a handful of world heritage sites in developed countries which are considered to be in danger, joining Liverpool's historic centre and docklands and Everglades National Park in the US.
Unesco is concerned about the impact of planned coastal developments, including ports and liquefied natural gas facilities, and said it had asked Australia to submit an updated report on the state of conservation of the site by February next year.
World Heritage Centre director Kishore Rao said the decision adopted by the World Heritage Committee, who are currently meeting in Doha, Qatar, welcomed progress made by Australia in managing the reef.
"Unesco is confident the overall direction towards next year's decision is a positive one," he said.
But environmental charity WWF-Australia said its government had been put on notice that it had to take stronger action to protect the reef, which supports more than 60,000 jobs and brings in billions of pounds a year in tourism revenue.
WWF-Australia reef campaigner Richard Leck said: "The Australian and Queensland governments must take responsibility, lift their game and improve management of the Reef.
"If not they face the shame of having the Reef declared 'World Heritage in Danger' in 2015, putting at risk thousands of tourism jobs.
"Australia has clearly not lived up to the standards expected by the international community when it comes to protecting the Reef."
Campaigners welcomed the move by RBS to rule out investment in the Abbot Point project.
Felicity Wishart, Great Barrier Reef campaigner for the Australian Marine Conservation Society, said: "While the Australian and Queensland governments have approved millions of tonnes of destructive dredging and dumping at Abbot Point, it is heartening to see global financial institutions such as the RBS take leadership and reject investment in projects that will damage the Reef."
"The health of the Great Barrier Reef is already suffering due to poor water quality.
"Port expansions would mean millions of tonnes of seabed being dredged and dumped in the Reef's waters - sediment would cloud the water, damaging seagrass and coral, making matters far worse."
Ben Pearson, of Greenpeace, said: "It is great to see that three global investment banks have now publicly distanced themselves from this risky, irresponsible project.
"It will be interesting to see if other British banks follow their lead over the coming months - particularly in the light of the strong concern that Unesco have expressed around the port projects."
The Great Barrier Reef covers an area of 348,000 square kilometres (133,000 square miles) along the north-eastern coastline of Australia and has been a world heritage site since 1981.
It is home to 400 types of coral, 1,500 species of fish and 4,000 types of mollusc.