Trams began running in the city on Saturday after years of delays, spiralling costs and a lengthy dispute between the city council and its contractor.
Overall, the controversial scheme to return trams to the streets of Scotland's capital has seen the construction of a line from Edinburgh Airport to York Place, costing about £776 million.
Repeated calls for a public inquiry have now been answered by the Scottish Government.
Speaking at First Minister's Questions at Holyrood, Mr Salmond said: "We cannot lose sight of the considerable public concern over the conduct of the project, the disruption it has caused to households and businesses in the city of Edinburgh.
"I therefore recommended to the Cabinet, and it has been decided, to establish a judge-led public inquiry into the Edinburgh trams project to establish why the project occurred significant overruns in terms of cost and timing, requiring in particular a considerable reduction in the original scope.
"It is important that there are lessons to be learnt from the conduct of the Edinburgh trams project."
Edinburgh Central MSP Marco Biagi welcomed the inquiry.
He said: "All of us who opposed the tram project from the start as risky and over-engineered have been disappointed almost daily to be shown be right.
"Does the First Minister agree with me that now that the trams are rolling, if there is to be any faith from the public in future management or potential cost estimates for projects like this, we need to know for sure these mistakes will never be repeated?"
Mr Salmond said: "The decision that we have made is to have a non-statutory inquiry."
He said this was due to timescale and because the transport minister had been assured by the council of full co-operation and full documentation of all aspects of the project.
"That gives of the opportunity to have a judge-led inquiry which will give us a proper examination and a public account of what has happened to the trams project," he said.
The new tram network was initially intended to be larger than it is now, stretching to the waterfront at Leith and Newhaven, and it was originally hoped to be in operation in 2010.
It was finally delivered three years late, about £230 million over-budget and reduced to a single route linking the city centre and the airport.
Many Edinburgh residents grew increasingly frustrated with the project as they had to put up with dug-up roads and traffic congestion while others remain angry at the cost.
Transport minister Keith Brown will give a statement to Parliament on the trams inquiry before the summer recess.
The director of national sustainable transport organisation Transform Scotland Colin Howden said it was "disappointing that the inquiry will not cover the cost escalation that is rampant across the other transport infrastructure projects that the Scottish Government manages".
He said: "The urban M74 ended up costing almost three times the cost at which it was initially approved by Scottish ministers.
"Meanwhile, the Aberdeen western bypass project is currently projected to cost six times the price at which it was approved by ministers.
"We would welcome similar levels of political scrutiny of the rest of the Scottish Government's infrastructure investment plans."
Labour's transport spokesman Mark Griffin said: "I welcome the announcement that there will be an independent public inquiry into the development of the Edinburgh tram line.
"Given the massive cost to the taxpayer, far in excess of what it should have cost, the impact on residents and businesses in Edinburgh and the long-term ongoing financial cost to Edinburgh, this decision is right.
"Taxpayers, businesses and the people of Edinburgh need to understand what went wrong and how this can never happen again.
"This is not the first or only major public contract which has not run to time or budget.
"I hope that this inquiry provides the lessons we need to ensure that the way in which we contract, organise and deliver these projects in the future means that this never happens again."