A joint report by the Higher Education Academy Scotland (HEA) and student body NUS Scotland found so-called "direct entry" students were most likely to describe settling into university life as challenging.
They found particular difficulties fitting in with other students who had already established social networks and adapting to new ways of learning and teaching.
These students, who tend to be from more deprived backgrounds, reported feeling less confident in playing an active role in their education and were less likely to feel they were achieving their full potential.
Overall, the majority of students had positive experiences in education, but the study also found student choices were dictated by negative perceptions of either university or college.
The difficulties surrounding direct entry in the second or third year of a degree are concerning because it is a key part of the Scottish Government's strategy of widening access to bright students from deprived communities.
Students with no experience of university can feel more comfortable at a local college studying HND courses which lead on to a degree rather than going straight to university and the Government has funded extra places through this route.
However, these students can be more at risk of dropping out of their university courses than those who are either better off or who have more family support.
The report recommends further work to support students from more deprived backgrounds and to help with their transition between levels of study, particularly for direct-entry students.
Robert Foster, vice-president of NUS Scotland, said: "We need to look more closely at why certain groups of students are facing more difficulties in education or are feeling left behind.
"We need to work together across the sectors to improve support for these students, particularly as we are working to increase the numbers accessing education from more disadvantaged backgrounds through direct-entry routes to university."
Mr Foster said it was also of concern that students made choices based on the negative perceptions of students in different parts of the further and higher education.
The report found college students felt university would be impersonal, difficult and intimidating, while university students saw college as a back-up plan or somewhere they would go if they did not achieve the grades for university.
Professor Stephanie Marshall, chief executive at the HEA, said: "It is encouraging to hear the majority of students surveyed are enjoying a positive experience, but the report highlights the need to widen access to education and ensure students have the necessary information to make the right choices."
The report's findings are based on a survey of more than 1600 college, undergraduate and postgraduate students in Scotland.
The HEA is a UK wide body, funded by the taxpayer as well as through subscriptions and grants, which works to improve the quality of universities.