Economists from Stirling University found it would be difficult to create a more equal society, such as Denmark, Iceland or Norway, by simply raising taxes for the better off and increasing welfare payments.
They warned high earners would leave Scotland if they faced significantly higher taxes than in the UK and suggested inequality could only be reduced to Nordic levels by "pre-distributive" measures to tackle the gulf between people's basic incomes.
The findings were reported in a new study, Constitutional Change and Inequality in Scotland, published as part of the Economic and Social Research Council's Future of the UK and Scotland project.
The report follows repeated claims by SNP ministers that reversing the UK's high levels of inequality could only be achieved under independence.
It showed the UK has the seventh highest level of income inequality in a league table of 35 developed countries. Scotland is mid-table, in 18th place, as the UK's position is skewed by high levels of inequality in London.
However, Scotland still lags significantly behind the four most equal countries, Iceland, Slovenia, Norway and Denmark, based on the GINI figure which measures differences in net incomes.
The Stirling study found a range of redistributive policies available to an independent Scotland - or to a devolved administration with increased tax powers - would fail to close the gap between Scotland and the likes of Denmark.
It said a 1p increase in the upper rate of income tax would only close the gap by 1%; while cutting the 50% tax threshhold from £150,000 to £100,000 would close the gap by 1.6%. A 10% increase in jobseekers' allowance and income support would only close the gap by 3.2%, the report found.
Other measures, such as raising council tax or increasing the Scottish rate of income tax due to be introduced in 2016, would also have only a marginal impact.
It said: "Labour mobility is high within countries, even if they are decentralised, and it would likely also be high in the early years of an independent state.
"It is difficult for a small open economy to implement substantially different redistributive policy from that of its close partners."
Dr David Comerford, one of the report's authors, said: "An independent Scotland would have access to fiscal powers with which it could influence inequality more directly than it can at the moment.
"However, achieving the level of inequality reduction the Scottish Government desires through fiscal policy alone would require major policy change."
Co-author David Eiser said: "A fiscal solution to inequality could be detrimental to government finances because raising taxes will increase incentives for high earners to relocate, and raising benefits could harm work incentives.
"Achieving Nordic levels of inequality in Scotland will likely have to involve some equalisation of incomes before taxes and benefits, rather than a large increase in redistribution."
Responding, Deputy First Minister Nicola Sturgeon said: "This report makes clear that Scotland currently has limited powers to properly tackle the huge inequality which has seen the UK become one of the most unequal societies in the developed world.
"It also finds that independence will give a Scottish Government real economic and social policy tools to address the issue."