The document will say the proposals, under which EU students would go free, would be discriminatory.
Foreign Secretary William Hague and Chief Secretary to the Treasury Danny Alexander will launch the paper, the latest in a Whitehall series, in Glasgow on Friday.
Focused on world affairs, it is expected to trigger another war of words between London and Edinburgh over an independent Scotland's membership of international bodies such as the European Union.
The Scottish Government has said that students from the rest of the UK would continue to pay tuition fees if Scotland became independent, while those from other EU countries would not.
A Whitehall source said: "The paper will confirm the EU legal position on tuition fees, that what the Scottish Government propose would be illegal under EU law.
"You cannot discriminate between EU member states."
A Scottish Government spokesman said: "Our policy is based on the unique and exceptional position of Scotland in relation to other parts of the UK, on the relative size of the rest of the UK, on the fee differential, on our shared land border and common language, on the qualification structure, on the quality of our university sector and on the high demand for places.
"We believe these distinctive characteristics will enable us to justify objectively the continuation of our current policy in a way which is consistent with the principles of free movement across the EU as a whole and which is compatible with EU requirements."