Shetland Line (1984) Ltd claims Transport Scotland broke tendering rules when it made its decision.
The row is about the so-called "standstill period" between deciding which bid to accept and concluding a deal with the successful firm.
The delay - based on a European directive - is intended to give other parties a chance to review the decision and mount a challenge.
The regulations also spell out a need for an unsuccessful bidder to inform a public body awarding a contract before raising a court action.
The Court of Session in Edinburgh heard about an exchange of e-mails between Shetland Line and Transport Scotland in May 2012.
Lawyers for Transport Scotland said the messages from Shetland Line did not meet the notification requirements.
Judge Lord Woolman rejected their attempt to get the case thrown out, ruling the firm had satisfied the regulations. A further hearing is likely at a later date.
In his ruling, Lord Woolman noted Shetland Line had given up on any hope of running the Aberdeen-Kirkwall-Lerwick ferries but was still seeking damages.
The £243 million contract was awarded to Serco.