Scotland currently gives fee-free education to Scots but charges students from England, Wales and Northern Ireland.
However, Edinburgh University researchers investigating attitudes to tuition fee policies found that "respondents believed that this arrangement was unlikely to continue if Scotland became an EU member state following a vote for independence".
They interviewed 50 higher education policymakers and other stakeholders in Scotland and the rest of the UK and 148 young people aged 14 to 19 in schools in Scotland and the north of England. Experts also analysed data from the Higher Education Statistics Agency.
The Scottish Government insists it will find a loophole to EU regulations which require member states to provide an equal level of funding for foreign EU students.
It says it has an "objective justification" for charging fees because the high fees south of the border would see an independent Scotland overwhelmed by applications from rUK students seeking cheaper education.
But Edinburgh University found "concern about an influx of students from England choosing free education in Scotland to avoid paying fees at home is unfounded".
Professor Sheila Riddell, of the University of Edinburgh's Moray House School of Education, said: "It is quite clear that whatever the outcome of the referendum in September, policy on higher education tuition fees in Scotland and England will continue to be very high on the political agenda.
"Whilst UK higher education is very successful, all countries face major challenges in relation to the funding and stability of the sector.
"The HE systems in the four countries of the UK are tightly inter-meshed and decisions made in one country have major knock-on consequences for the others."
A Scottish Government spokeswoman said: "The Scottish Government is firmly committed to free tuition and we need to ensure that there are enough funded places for Scottish-domiciled students. This is important for them, their families and the economy.
"In 2012/13, 1.5% of English-domiciled full-time first degree students studied in Scottish higher education institutions, but these students accounted for 10.7% of full-time first degree students at these institutions.
"Even a limited increase in the proportion of students from England could have a significant impact on access to places for Scottish-domiciled students.
"The ability to benefit from free tuition would provide a very strong incentive for students being asked to pay up to £27,000 over three years to come to Scotland in increased numbers. That is why our policy on this matter is in place.
"If the UK Government was to reconsider its policy on charging tuition fees, of course the Scottish Government would reconsider in light of any changes."
Scottish Conservative education spokeswoman Mary Scanlon said: "This is yet another example of experts lining up to dismiss more claims from the SNP's white paper.
"We've always known that European law could not be clearer on this issue.
"An independent Scotland would not be able to discriminate between different national groups and that has been confirmed in this in-depth study.
"It also seems Mike Russell's warning about a flood of students from the rest of the UK fleeing here to avoid paying student fees is completely wide of the mark."
Liberal Democrat education spokesman Liam McArthur said: "There are currently over 4,800 undergraduate students from the rest of the UK paying fees to study in Scotland.
"Even if there was not a significant increase in students from the rest of the UK in the event of a vote for independence, there would still be a significant funding gap to fill.
"You don't need a degree in maths to see that these sums just don't add up."