Alistair Carmichael claimed the Scottish Government's proposed route to EU membership has never before been used to expand the organisation.
The Scottish Secretary said: "The process they propose is flawed in legal terms and destined to fail in the cold hard light of political reality."
He also accused the SNP administration at Holyrood of having "failed to tell Scots the reality" of this.
Mr Carmichael, who was speaking in Brussels, argued that an independent Scotland would have to apply for EU membership "in the same way that every other new member has", using Article 49 of the Treaty of the European Union.
To achieve this, he said Scotland would have to win the "unanimous support" of the European Council, have its membership approved through an accession treaty and then have its application ratified by each existing member state.
The Scottish Government's white paper, however, argues it would be ''appropriate'' for an independent Scotland to use Article 48 of the Treaty of the European Union to gain membership - adding countries would only use Article 49 to achieve this status if they were outside the EU.
Mr Carmichael said: "We are told by the Scottish Government in their white paper that Scotland could become a new member of the EU by the 'back door'.
"The so-called 'Article 48 route' is held up by the Scottish Government as the super highway to EU membership."
But he added: "In reality, this is a dead end. Article 48 has never been used to expand membership of the EU.
"There is no way round the law. A new state must apply, it would be no different for an independent Scotland."
He also claimed the Scottish Government-proposed independence timetable would weaken Alex Salmond and the Scottish Government's negotiating position with the EU.
Nationalists propose if there was a Yes vote in the referendum in September, Scotland could become an independent country 18 months later, in March 2016.
The Scottish Secretary branded this timetable "a negotiating position of extraordinary weakness", adding: "One man's obsession to deliver independence not just to a specific timetable, but to a specific day of the week...would not just undermine Alex Salmond's hand in negotiations, but Scotland's future in Europe."
He criticised the First Minister and said: "Instead of showing he has Scotland's interests at heart, this obsession with a date rather than the deal reveals just how much of a vanity project this really is."
Mr Carmichael stressed if Scotland left the UK, it would need to negotiate its EU membership, and "critically" the terms of that, with all 28 EU member states.
He insisted that by being part of the UK Scots not only "gain from our status as European citizens" but also "benefit still further from being one of the largest member states in the EU".
His attack came days after a UK Government paper last week said that being independent within Europe would leave Scotland up to £3.8 billion worse off, as a result of the loss of the UK's EU rebate and different levels of payments from other EU funding schemes
The Scottish Secretary said today that the the budget rebate, together with the UK's opt-outs on joining the euro and the Schengen area, in which people can travel between countries without showing passports, would be "at risk" in Scotland if it became a new state.
He said: "All of these things that the Nationalists say they would want for Scotland in the EU - the exemption from the euro and Schengen, the retention of the rebate. All of which as part of the UK, Scotland already has today.
"Leaving the UK means leaving the EU, then trying to fight your way back in seeking the same terms from a weaker position. This runs against Scotland's interests."