Italian financier Andrea Orcel was part of the Merrill Lynch team that reportedly received a $500 million (£312m)fee for its work on the 2007 transaction.
The pressure placed on RBS's finances by the purchase has been blamed for the Edinburgh bank requiring a £45 billion taxpayer bailout a year later.
Mr Orcel, now chief executive of the investment bank at Swiss-based UBS, told the Parliamentary Commission on Banking Standards: "With what I know today we would have advised them not to proceed."
But he resisted blaming the takeover for RBS's financial problems and subsequent part-nationalisation. He said: "I think much will be written on what brought the bank down. There is different opinion on that specifically."
The Financial Services Authority later criticised the lack of due diligence conducted by the RBS board on the £49bn deal, saying its size and the lack of clarity on the risks involved made it a gamble.
In particular, it noted RBS's investment banking advisers were "largely remunerated on a success fee basis". This made it "difficult" to argue that the advice they gave was independent, the FSA found.
In his testimony to the commission, Mr Orcel, who has been involved in a number of large European banking mergers, denied he was a "deal junkie" and said he had worked with most of his clients for 15 to 20 years.
He said: "One client in the past said the reason he took my advice is that I say more times not to do things than to do things."
Mr Orcel came under particular scrutiny in 2008 when it was revealed he received a £21.3m bonus, at a time when most of the industry was cutting back.
UBS was fined £940m last month for manipulating Libor interest rates, was hit by a rogue-trading loss and fined in relation to clients' tax avoidance in the United States.
Mr Orcel said UBS had learned "some very tough lessons" from the problems.
But head of compliance Andrew Willaims admitted UBS has "terminated" just 18 of the 40 people at the centre of the Libor scandal which took place between 2006 and 2009.
Mr Orcel said "90-plus" per cent of the traders involved had been men and conceded that a higher proportion of female staff could change the culture.
"I think that is a shortcoming of UBS," he said.
He also said the bank would consider a proposal made by Lord McFall of Alcluith, the former MP for West Dunbartonshire, that UBS establish a culture and ethics hotline for unhappy staff, the contents of which would be reported to the board.
Lord McFall said: "All the evidence we have received so far in culture and ethics leads to the tone at the top."