The agency calculated RBS generated $1.2 billion (£740 million) fees for advising on mergers, share listings and bond issues in 2012, down 11.9% on the preceding year. The total revenues shared by all sector players for such work fell by 3.4% annually, to $74.8bn, the lowest level since 2009.
Thomson Reuters calculated the Royal Bank's market share fell by 0.2 percentage points to 1.6%.
Royal Bank has been cutting its exposure to investment banking under chief executive Stephen Hester as part of efforts to reduce its balance sheet and risk profile.
JP Morgan topped the table compiled by Thomson Reuters, although the US bank saw revenues fall by 2.2% annually, to $5.6bn. JP Morgan's market share increased by 0.1 percentage points to 7.5%.
Separately, Royal Bank of Scotland's Ulster Bank arm said it would close around 10% of its branches, in a move that reflects tough conditions in markets like property in Ireland.
The company said it will close around 20 branches and sub offices out of 226 split between the Republic of Ireland and Northern Ireland.